Simple encryption for people on a byte budget

138 bytes, machine code (16-bit x86)

000000 88 9f 00 02 fe c3 75 f8 b9 01 00 89 fa b4 3f cd
000010 21 86 0d ba 00 03 b4 3f cd 21 89 dd 01 f6 02 9e
000020 00 03 8a 04 00 c3 86 87 00 02 88 04 46 45 39 cd
000030 75 02 31 ed 81 fe 00 03 75 e4 81 ee 00 01 31 db
000040 31 d2 ff c6 81 fe 00 03 75 04 81 ee 00 01 8a 04
000050 00 c3 88 c2 86 87 00 02 88 04 00 c2 89 d7 81 c7
000060 00 02 8a 15 89 d7 89 dd 31 db ba 00 03 b9 01 00
000070 b8 00 3f cd 21 87 fa 30 15 85 c0 74 0b 87 fa 43
000080 b4 40 cd 21 89 eb eb b8 cd 20

Running: save to codegolf.com, dosbox:

codegolf.com < input.bin

successfull attempt

I don't know if this gonna count as an entry, but I've decided to do it manually using hex editors. No compilers were used to do this.

ht editor actually has assembler, but honestly I did not know about this until I was finished ¯\_(ツ)_/¯

Why & how

Why: mostly cause I wanted to check if I'm able to do this.

How: I've started with creating byte filled with NOPs and following with a simple part: trying to write first loop that fills State with 0..255 values. I switched to python and quickly wrote python version, just to have something to test against. Next I was simplifying python code into pseudo code / pseudo assembly. Then I I was trying to write small pieces. I decided it'll be easiest to read from stdin, so I started with something small that will read single byte, then I added password reading and key initialization. Figuring out what registers to pick took me some time.

I though adding de/encryption loop will be easy, but first I actually got single byte decode and added whole loop afterwards.

Last step was getting rid of additional nops that I've left between instructions when writing it (ofc that required fixing jumps as well).

You can see small gallery that I tried to make while progressing here.

Dissection

The program relies on some initial values after startup (see resources below).

00000000 889f0002                  mov         [bx+0200], bl
00000004 fec3                      inc         bl
00000006 75f8                      jnz         0x0

fill in State (at 0x200)

00000008 b90100                    mov         cx, 0x1
0000000b 89fa                      mov         dx, di
0000000d b43f                      mov         ah, 0x3f
0000000f cd21                      int         0x21
00000011 860d                      xchg        [di], cl
00000013 ba0003                    mov         dx, 0300
00000016 b43f                      mov         ah, 0x3f
00000018 cd21                      int         0x21

read length, read password, store password at ds:0x300

0000001a 89dd                      mov         bp, bx
0000001c 01f6                      add         si, si
0000001e 029e0003                  add         bl, [bp+0300]
00000022 8a04                      mov         al, [si]
00000024 00c3                      add         bl, al
00000026 86870002                  xchg        [bx+0200], al
0000002a 8804                      mov         [si], al
0000002c 46                        inc         si
0000002d 45                        inc         bp
0000002e 39cd                      cmp         bp, cx
00000030 7502                      jnz         0x34
00000032 31ed                      xor         bp, bp
00000034 81fe0003                  cmp         si, 0300
00000038 75e4                      jnz         0x1e

initialize State with a key (BP is used to traverse key, SI is used to traverse State)

0000003a 81ee0001                  sub         si, 0100
0000003e 31db                      xor         bx, bx
00000040 31d2                      xor         dx, dx
00000042 ffc6                      inc         si
00000044 81fe0003                  cmp         si, 0300
00000048 7504                      jnz         0x4e
0000004a 81ee0001                  sub         si, 0100
0000004e 8a04                      mov         al, [si]
00000050 00c3                      add         bl, al
00000052 88c2                      mov         dl, al
00000054 86870002                  xchg        [bx+0200], al
00000058 8804                      mov         [si], al
0000005a 00c2                      add         dl, al
0000005c 89d7                      mov         di, dx
0000005e 81c70002                  add         di, 0200
00000062 8a15                      mov         dl, [di]

generate pseudo random value (in DL, DH is 0 thx to xor at 0x140)

00000064 89d7                      mov         di, dx      
00000066 89dd                      mov         bp, bx      
00000068 31db                      xor         bx, bx      
0000006a ba0003                    mov         dx, 0300    
0000006d b90100                    mov         cx, 0x1     
00000070 b8003f                    mov         ax, 3f00    
00000073 cd21                      int         0x21        
00000075 87fa                      xchg        dx, di      
00000077 3015                      xor         [di], dl    
00000079 85c0                      test        ax, ax      
0000007b 740b                      jz          0x88        
0000007d 87fa                      xchg        dx, di      
0000007f 43                        inc         bx          
00000080 b440                      mov         ah, 0x40    
00000082 cd21                      int         0x21        
00000084 89eb                      mov         bx, bp      
00000086 ebb8                      jmp         0x40        
00000088 cd20                      int         0x20        
  • store values that we need to preserve (SI - ints won't touch it, BX)
  • read char from input, xor it
  • quit if end of stream
  • output decoded char
  • restore values
  • loop to 0x40 (reuse xor on DX)

P.S. This probably could be even shorter, but this took 4 evenings, so not sure if I want to spend another one...

Tools and resources

  • ht editor - it has 16-bit disasm
  • hexplorer - mostly for it's mouse based copy/paste + insert/overwrite
  • dosbox - for debugging
  • X86 Opcode and Instruction Reference 1.12 and invaluable 16-bit ModR/M Byte
  • DOS INT 21h - DOS Function Codes
  • 80386 Instruction Set note this is 386, but that's more than enough
  • Register values at DOS .COM startup

C (gcc), 193 188 182 178 171 172 bytes

f(x,l)int*x;{unsigned char*X=x,i=0,j=0,S[256],t;for(;S[i]=++i;);for(;t=S[i],S[i]=S[j+=t+X[1+i%*X]],S[j]=t,t=++i;);for(X+=*X;l--;S[i]-=S[t]=j)*++X^=S[S[i]+=S[t+=j=S[++i]]];}

Try it online!

Edit: Now works with keys longer than 127 bytes.

Edit2: Added testcase with 129 byte key to TIO link.

Slightly less golfed version

f(x,l)int*x;{
  unsigned char*X=x,i=0,j=0,S[256],t;
  // initialize state
  for(;S[i]=++i;);
  // key processing
  for(;t=S[i],S[i]=S[j+=t+X[1+i%*X]],S[j]=t,t=++i;);
  // encrypt
  for(X+=*X;l--;S[i]-=S[t]=j)
    *++X^=S[S[i]+=S[t+=j=S[++i]]];
}

CPU x86 instruction set, 133 bytes

000009F8  53                push ebx
000009F9  56                push esi
000009FA  57                push edi
000009FB  55                push ebp
000009FC  55                push ebp
000009FD  BF00010000        mov edi,0x100
00000A02  29FC              sub esp,edi
00000A04  8B6C3C18          mov ebp,[esp+edi+0x18]
00000A08  31DB              xor ebx,ebx
00000A0A  8A5D00            mov bl,[ebp+0x0]
00000A0D  45                inc ebp
00000A0E  31C0              xor eax,eax
00000A10  880404            mov [esp+eax],al
00000A13  40                inc eax
00000A14  39F8              cmp eax,edi
00000A16  72F8              jc 0xa10
00000A18  31F6              xor esi,esi
00000A1A  31C9              xor ecx,ecx
00000A1C  89F0              mov eax,esi
00000A1E  31D2              xor edx,edx
00000A20  F7F3              div ebx
00000A22  8A0434            mov al,[esp+esi]
00000A25  02441500          add al,[ebp+edx+0x0]
00000A29  00C1              add cl,al
00000A2B  8A0434            mov al,[esp+esi]
00000A2E  8A140C            mov dl,[esp+ecx]
00000A31  88040C            mov [esp+ecx],al
00000A34  881434            mov [esp+esi],dl
00000A37  46                inc esi
00000A38  39FE              cmp esi,edi
00000A3A  72E0              jc 0xa1c
00000A3C  8B443C1C          mov eax,[esp+edi+0x1c]
00000A40  01E8              add eax,ebp
00000A42  722F              jc 0xa73
00000A44  48                dec eax
00000A45  89C6              mov esi,eax
00000A47  01DD              add ebp,ebx
00000A49  31C0              xor eax,eax
00000A4B  31D2              xor edx,edx
00000A4D  31C9              xor ecx,ecx
00000A4F  39F5              cmp ebp,esi
00000A51  7320              jnc 0xa73
00000A53  FEC2              inc dl
00000A55  8A0414            mov al,[esp+edx]
00000A58  00C1              add cl,al
00000A5A  8A1C0C            mov bl,[esp+ecx]
00000A5D  88040C            mov [esp+ecx],al
00000A60  881C14            mov [esp+edx],bl
00000A63  00D8              add al,bl
00000A65  8A1C04            mov bl,[esp+eax]
00000A68  8A4500            mov al,[ebp+0x0]
00000A6B  30D8              xor al,bl
00000A6D  884500            mov [ebp+0x0],al
00000A70  45                inc ebp
00000A71  EBDC              jmp short 0xa4f
00000A73  01FC              add esp,edi
00000A75  5D                pop ebp
00000A76  5D                pop ebp
00000A77  5F                pop edi
00000A78  5E                pop esi
00000A79  5B                pop ebx
00000A7A  C20800            ret 0x8
00000A7D

A7D-9F8=85h=133 bytes but i don't know if calculation is ok because the preciding number of bytes of the same function result 130 bytes... The first argumenti of the function that I name "cript" is the string, the second argumenti is the string lenght (first byte+ key lenght + message lenght). Below there is the assembly language file for obtain that cript routines:

; nasmw -fobj  this.asm

section _DATA use32 public class=DATA
global cript
section _TEXT use32 public class=CODE

cript:    
      push    ebx
      push    esi
      push    edi
      push    ebp
      push    ebp
      mov     edi,  256
      sub     esp,  edi
      mov     ebp,  dword[esp+  edi+24]
      xor     ebx,  ebx
      mov     bl,  [ebp]
      inc     ebp
      xor     eax,  eax
.1:   mov     [esp+eax],  al
      inc     eax
      cmp     eax,  edi
      jb      .1
      xor     esi,  esi
      xor     ecx,  ecx
.2:   mov     eax,  esi
      xor     edx,  edx
      div     ebx
      mov     al,  [esp+esi]
      add     al,  [ebp+edx]
      add     cl,  al
      mov     al,  [esp+esi]
      mov     dl,  [esp+ecx]
      mov     [esp+ecx],  al
      mov     [esp+esi],  dl
      inc     esi
      cmp     esi,  edi
      jb      .2
      mov     eax,  dword[esp+  edi+28]
      add     eax,  ebp
      jc      .z
      dec     eax
      mov     esi,  eax
      add     ebp,  ebx
      xor     eax,  eax
      xor     edx,  edx
      xor     ecx,  ecx
.3:   cmp     ebp,  esi
      jae     .z
      inc     dl
      mov     al,  [esp+edx]
      add     cl,  al
      mov     bl,  [esp+ecx]
      mov     [esp+ecx],  al
      mov     [esp+edx],  bl ; swap S[c] S[r]
      add     al,  bl
      mov     bl,  [esp+eax]
      mov     al,  [ebp]
      xor     al,  bl
      mov     [ebp],  al
      inc     ebp
      jmp     short  .3
.z:       
      add     esp,  edi
      pop     ebp
      pop     ebp
      pop     edi
      pop     esi
      pop     ebx
      ret     8

below the C file for check results:

// Nasmw  -fobj  fileasm.asm
// bcc32 -v filec.c fileasm.obj
#include <stdio.h>

void _stdcall cript(char*,unsigned);

char es1[]="\x01\x00\x00\x00\x00\x00\x00";
char es2[]="\x0Dthis is a keythis is some data to encrypt";
char es3[]="\x0dthis is a key\xb5\xdb?i\x1f\x92\x96\226e!\xf3\xae(!\xf3\xea\x43\xd4\x9fS\xbd?d\x82\x84{\xcdN";
char es4[]="Sthis is a rather long key because the value of S is 83 so the key length must matchand this is the data to be encrypted";

void printMSGKeyC(unsigned char* a, unsigned len)
{unsigned i,j,k;
 unsigned char *p,*end;

 printf("keylen = %u\nKey    = [", (unsigned)*a);
 for(i=1, j=*a;i<=j;++i) printf("%c", a[i]);
 printf("]\nMessage= [");
 for(p=a+i,end=a+len-1;p<end;++p)printf("%c", *p);
 printf("]\n");
}

void printMSGKeyHex(unsigned  char* a, unsigned len)
{unsigned i,j,k;
 unsigned char *p,*end;

 printf("keylen = %u\nKey    = [", (unsigned)*a);
 for(i=1, j=*a;i<=j;++i) printf("%02x", a[i]);
 printf("]\nMessage= [");
 for(p=a+i,end=a+len-1;p<end;++p)printf("%02x", *p);
 printf("]\n");
}


main()
{printf("sizeof \"%s\"= %u [so the last byte 0 is in the count]\n", "this", sizeof "this");

 printf("Input:\n");
 printMSGKeyHex(es1, sizeof es1);
 cript(es1,  (sizeof es1)-1);
 printf("Afther I cript:\n");
 printMSGKeyHex(es1, sizeof es1);

 printf("Input:\n");
 printMSGKeyC(es2, sizeof es2);
 printMSGKeyHex(es2, sizeof es2);
 cript(es2,  (sizeof es2)-1);
 printf("Afther I cript:\n");
 printMSGKeyC(es2, sizeof es2);
 printMSGKeyHex(es2, sizeof es2);
 cript(es2,  (sizeof es2)-1);
 printf("Afther II cript:\n");
 printMSGKeyC(es2, sizeof es2);
 printMSGKeyHex(es2, sizeof es2);
 printf("----------------------\n");

 printf("Input:\n");
 printMSGKeyHex(es3, sizeof es3);
 cript(es3,  (sizeof es3)-1);
 printf("Afther I cript:\n");
 printMSGKeyHex(es3, sizeof es3);
 printf("----------------------\n");

 printf("Input:\n");
 printMSGKeyHex(es4, sizeof es4);
 cript(es4,  (sizeof es4)-1);
 printf("Afther I cript:\n");
 printMSGKeyHex(es4, sizeof es4);
 cript(es4,  (sizeof es4)-1);
 printf("Afther II cript:\n");
 printMSGKeyHex(es4, sizeof es4);

 return 0;
}

the results:

 sizeof "this"= 5 [so the last byte 0 is in the count]
Input:
keylen = 1
Key    = [00]
Message= [0000000000]
Afther I cript:
keylen = 1
Key    = [00]
Message= [de188941a3]
Input:
keylen = 13
Key    = [this is a key]
Message= [this is some data to encrypt]
keylen = 13
Key    = [746869732069732061206b6579]
Message= [7468697320697320736f6d65206461746120746f20656e6372797074]
Afther I cript:
keylen = 13
Key    = [this is a key]
Message= [Á█?iÆûûe!¾«(!¾ÛCȃS¢?déä{═N]
keylen = 13
Key    = [746869732069732061206b6579]
Message= [b5db3f691f9296966521f3ae2821f3ea43d49f53bd3f6482847bcd4e]
Afther II cript:
keylen = 13
Key    = [this is a key]
Message= [this is some data to encrypt]
keylen = 13
Key    = [746869732069732061206b6579]
Message= [7468697320697320736f6d65206461746120746f20656e6372797074]
----------------------
Input:
keylen = 13
Key    = [746869732069732061206b6579]
Message= [b5db3f691f9296966521f3ae2821f3ea43d49f53bd3f6482847bcd4e]
Afther I cript:
keylen = 13
Key    = [746869732069732061206b6579]
Message= [7468697320697320736f6d65206461746120746f20656e6372797074]
----------------------
Input:
keylen = 83
Key    = [74686973206973206120726174686572206c6f6e67206b65792062656361757365207468652076616c7565206f66205320697320383320736f20746865206b6579206c656e677468206d757374206d61746368]
Message= [616e64207468697320697320746865206461746120746f20626520656e63727970746564]
Afther I cript:
keylen = 83
Key    = [74686973206973206120726174686572206c6f6e67206b65792062656361757365207468652076616c7565206f66205320697320383320736f20746865206b6579206c656e677468206d757374206d61746368]
Message= [961f2c8fa3259ba3665b6d6bdfbcac8b8efafe96423d21fc3b13606316710411d886ee07]
Afther II cript:
keylen = 83
Key    = [74686973206973206120726174686572206c6f6e67206b65792062656361757365207468652076616c7565206f66205320697320383320736f20746865206b6579206c656e677468206d757374206d61746368]
Message= [616e64207468697320697320746865206461746120746f20626520656e63727970746564]