Pi and PIC graphics formats
| Japanese | English |
|---|---|
| Pi tech | Pi tech |
| PIC format | PIC format |
Various Japanese specifications and native X68000 software can be found here. The table on the right links to the specifications.
The open-source Recoil project specialises in handling many old image formats such as these. They have a handy browser-based viewer here.
Grapholic and Susie may be able to natively view Pi and PIC files. The ViX tool for Win9x/XP can also view and convert Pi and PIC files.
My own code for opening these files is on Gitlab as part of SuperSakura. You can convert Pi and PIC files to PNG with the "decomp" tool. There is a win32 binary of the whole project here, which contains decomp.
Background
The Pi and PIC formats were developed around 1990 by the clever programmer Yanagisawa-san, of whom I can't find any information apart from the name. PIC was targeted at the high-color-capable X68000 computers, while Pi was optimal for the reduced palette used by the PC98.
Although Pi uses a strictly superior algorithm compared to MAG, the latter's slight headstart had already established it as the standard file format for artists and fans sharing images on BBSes. It also can't have hurt that the developer of the MAG format also created the most popular PC98 image editor. And in a few years, GIF with its on average even better LZW compression took the crown, while JPGs eclipsed PIC and PIC2. (Also, don't confuse these with Apple's PICT format, which is a whole other thing.)
The Pi format did see use by Japanese software houses, who needed the most efficient available image compression to fit their games on fewer diskettes. I reverse-engineered the format without any documentation back around 2006, having grown curious about classic visual novel engines. Curiously, although clearly many others have written Pi decoders, there's still a dearth of English specifications, so I'm putting mine out, with additional detail gained from the original Japanese specifications.
Pi Specification
Pi header
(MSB-first, X68000 native)
| Offset | Size | Description |
|---|---|---|
| 0 | 2 | Signature "Pi" (sometimes omitted or zeroed out) |
| 2 | .. | Comment and metadata section, encoded in Shift-JIS; Variable-length, terminates with byte $1A, and the first 00 after $1A marks the start of the real header |
| 0 | 1 | Start of header, always 00 |
| 1 | 1 | Mode byte, usually 00 |
| 2 | 1 | Pixel aspect ratio X |
| 3 | 1 | Pixel aspect ratio Y |
| 4 | 1 | Bitdepth |
| 5 | 4 | Compressor model signature |
| 9 | 2 | Size of compressor-specific data, usually 0 |
| 11 | .. | Compressor-specific data section |
| .. | 2 | Image pixel width |
| .. | 2 | Image pixel height |
| .. | .. | Palette: 16 or 256 byte triplets, order RGB |
The mode byte's top bit is supposed to indicate that a default palette be used, but even my test images with a $FF mode still include the palette. You can maybe ignore this.
The pixel aspect ratio X:Y should signal whether any image stretching will be necessary after decompressing, but none of my test images make use of this feature. The ratios 0:0 and 1:1 both indicate a normal 1:1 pixel aspect ratio.
Bitdepth is 4 for 16-color images, and 8 for 256-color images. You'll need this to know how much palette data follows the header. If bitdepth is $FF, try defaulting to 16 colors.
The compressor model can be ignored, except to take into account how many colors the source system was capable of displaying. The X68K has 5 significant bits in its palette components, while the default PC98 systems have 4.
Compressor-specific data is hardly ever used, and I don't know how to interpret it. Doesn't seem to affect the end result, so skip it.
Every palette component byte should have its most significant bits copied to the rest of the byte, much like the MAG format does. The visual difference is minor even if you use the palette bytes directly as 8-bit values.
Pi decompression
Click the image to download the example Pi file, to test your decompressor.
To start with, create an output buffer and a delta table. The algorithm will be using a combination of delta codes and sequence repetition to fill the output buffer.
The output buffer should be 1 byte per pixel, and the exact size specified in the header. The image's bitdepth makes no difference here.
The delta table is like an array of linked lists, showing for each color byte a list of most recently associated color bytes. At a palette size of 16 colors, the table must be 16x16 bytes; at 256 colors, it must be 256x256 bytes. The table must be initialised using this:
table[x, y] = (number of colors + x - y) modulo (number of colors)
- The values in the table are the color byte being moved to
- x is the color byte being moved from
- y is a sort order, where 0 has the color byte that most recently followed x
An 8-color table for example would look like this (note the reversed axes):
| x \ y | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 |
|---|---|---|---|---|---|---|---|---|
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 0 |
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 0 | 1 |
| 2 | 3 | 4 | 5 | 6 | 7 | 0 | 1 | 2 |
| 3 | 4 | 5 | 6 | 7 | 0 | 1 | 2 | 3 |
| 4 | 5 | 6 | 7 | 0 | 1 | 2 | 3 | 4 |
| 5 | 6 | 7 | 0 | 1 | 2 | 3 | 4 | 5 |
| 6 | 7 | 0 | 1 | 2 | 3 | 4 | 5 | 6 |
| 7 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
So, whenever you need to process a delta code:
- x = preceding output byte (use 0 at the very start of the image)
- y = delta code
- The output byte is table[x,y]
- Move the value from table[x,y] into table[x,0], shifting the rest of the row backward to make room. This way [x,0] always has the color byte that most recently followed x, and [x,1] has the second-most recent, and so on.
Example: From an initial state using the above 8x8 table:
| Input delta code sequence | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| 5 | 7 | 4 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 1 |
| Output color bytes | |||||||||||
| 3 | 4 | 0 | 0 | 3 | 4 | 0 | 0 | 3 | 4 | 0 | 0 |
| Delta table afterward | ||||||||
|---|---|---|---|---|---|---|---|---|
| x \ y | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 |
| 0 | 1 | 2 | 4 | 5 | 6 | 7 | 3 | 0 |
| 1 | (no change) | |||||||
| 2 | (no change) | |||||||
| 3 | 5 | 6 | 7 | 0 | 1 | 2 | 3 | 4 |
| 4 | 5 | 6 | 7 | 1 | 2 | 3 | 4 | 0 |
| 5 | (no change) | |||||||
| 6 | (no change) | |||||||
| 7 | (no change) | |||||||
As you can see, any repeating pixel patterns in the image will quickly reconfigure the delta table so the pattern's colors can be encoded using short, low-value delta codes.
In the compressed stream, delta codes are stored using variable bit lengths, with the smallest codes assigned the shortest lengths. You'll need to read the delta codes one bit at a time, until you have a valid code. There are two different encodings, one for 16-color images, the other for 256-color images. Use the right one indicated by the header's bit depth.
| 4-bit delta encoding | |
|---|---|
| Binary encoding | Decimal delta code range |
| 1x | 0-1 |
| 00x | 2-3 |
| 010xx | 4-7 |
| 011xxx | 8-15 |
| 8-bit delta encoding | |
|---|---|
| Binary encoding | Decimal delta code range |
| 1x | 0-1 |
| 00x | 2-3 |
| 010xx | 4-7 |
| 0110xxx | 8-15 |
| 01110xxxx | 16-31 |
| 011110xxxxx | 32-63 |
| 0111110xxxxxx | 64-127 |
| 0111111xxxxxxx | 128-255 |
In the compressed stream, delta codes alternate with sequence repetition commands. Each repetition command has a relative location to repeat output from, followed immediately by a repetition length. The length is specified in byte pairs, so for example length 1 = 2 bytes, and 8 = 16 bytes.
There are five possible location codes, each of which is two or three bits long, as follows:
- 00: Repeat the last 4 output bytes, unless the last two bytes are equal or there are less than 4 output bytes so far, in which case repeat the last 2 output bytes.
- 01: Copy from exactly one row above. If trying to copy from before the start of the output buffer, keep copying the buffer's first two bytes.
- 10: Copy from exactly two rows above. Again, keep copying the buffer's first two bytes while out of bounds.
- 110: Copy from one row above, and one byte ahead. If trying to copy from before the start of the output buffer, keep copying the buffer's first two bytes in reverse order.
- 111: Copy from one row above, and one byte back. Again, keep copying the buffer's first two bytes, reversed, while out of bounds.
Repetition lengths are encoded with bit sequences like this:
| Repetition length encoding | |
|---|---|
| Binary encoding | Decimal output range |
| 0 | 1 |
| 10x | 2-3 |
| 110xx | 4-7 |
| 1110xxx | 8-15 |
| 11110xxxx | 16-31 |
| ... | ... |
All repetition commands are assumed to be immediately followed by another repetition command, unless the next repetition location code is exactly the same as the immediate prior location code.
Example repetition sequence:
| Input repetition commands | ||||
|---|---|---|---|---|
| Location | Length | Location | Length | Location |
| 110 | 11010 | 01 | 100 | 01 |
| Meaning | ||||
| Copy 6x2 bytes from one row above and one byte ahead | Copy 2x2 bytes from one row above | No more repeats | ||
Be sure to guard against repetition lengths going beyond the end of your output buffer. I have some test images that will crash an unguarded decoder.
To decompress an image bitstream, follow these steps:
- Process two delta codes.
- Process a repetition sequence, except with the first repeat's length decreased by two bytes.
- Then, until the output buffer is full:
- Process two delta codes.
- Read one bit. If it is set, go back to step 1.
- Process a repetition sequence, then go back to step 1.
Example bitstream from the beginning of the below 16-color image:
| Input stream (hexadecimal) | |
|---|---|
| 5D 29 80 26 75 | |
| Input stream (binary) | |
| 0101-1101 0010-1001 1000-0000 0010-0110 0111-0101 | |
| Input codes | Action |
| 01011 | Delta code 7: output color 9 (jade green) |
| 10 | Delta code 0: output color 9 |
| 10 | Repeat from two rows above... |
| 0 | ... 1x2 bytes, except this is the first repeat, so it's 0x2 bytes (nothing) |
| 10 | No more repeats |
| 10 | Delta code 0: output color 9 |
| 011000 | Delta code 8: output color 1 (burnt umber) |
| 0 | Going to repeat... |
| 00 | ... the last 4 output bytes... |
| 0 | ... for 1x2 bytes: output color 9 and 9 |
| 00 | No more repeats |
| 10 | Delta code 0: output color 1 |
| 011001 | Delta code 9: output color 8 (dark teal) |
| 1 | More delta codes follow... |
| 10 | Delta code 0: output color 8 |
| 10 | Delta code 0: output color 8 |
| 1 | More delta codes follow... |
PIC Specification
tbd