Understanding Disk Images by building a macOS Catalina ISO image for VirtualBox [macOS] Dec 4 2019 Latest Update: May 16 2020

Having a virtual machine running macOS opens up a lot of opportunities for learning. If you are into security, you can set up a VM for your security lab. Or if you want to learn networking or kernel debugging, it is also helpful to use a VM. The other option is risk breaking your work machine in your experiments (not fun). To build our VM, we need to use Disk Images, another topic that is useful in other areas.

Both topics are interesting, and we can combine them. In this post, I'll show you the basics of working with Disk Images and also as an example we are going to create an ISO image to build our macOS Catalina VM using VirtualBox.

There's a lot of information so let's get started.

NOTE: We need to download the macOS Catalina installer from the AppStore, it's around 9Gb, so It'll take some time to download. Better start now while we learn about Disk Images.

Table of Contents

Creating and working with disk images

We'll learn by making, so let's start by creating a folder that we'll use as the base for our image.

1

$ mkdir ~/Desktop/myapp

Let's add some content to our folder:

1

$ echo "Hola, mundo!" > ~/Desktop/myapp/hello.txt

Ok, this is going to be our simple image structure:

1
2

myapp/
└── hello.txt

We are going to use hdiutil. This tool uses the DiskImages.framework for... well, manage disk images. Or at least that is what its man page says. But I couldn't find any documentation on the DiskImages.framework so I assume it's a private framework. If you know anything about it, let me know. In the meantime, I encourage you to read the man page hdiutil(1) for more information.

1

$ hdiutil create -srcfolder ~/Desktop/myapp ~/Desktop/miapplicacion.dmg

Alright, that should have created miapplicacion.dmg image in your desktop. We can now delete the ~/Desktop/myapp, we don't need it anymore.

1

$ rm -r ~/Desktop/myapp

We can now mount the image. We can do that by clicking it using the Finder, or by using hdiutil again. I'll use hdiutil so we get more comfortable using it.

1

$ hdiutil attach ~/Desktop/miapplicacion.dmg

That will mount our image in /Volumes/myapp. I purposefully gave different names to the dmg and the image so you can see which one will be used when mounted.

If you check the newly mounted image, you'll see our hello.txt with the content Hola, mundo!.

Now you can eject it from the Finder or use hdiutil again. As you probably imagined, I'll show you how to do it with hdiutil.

1

$ hdiutil detach /Volumes/myapp

You can use the name of the volume or the device, in my case:

1

$ hdiutil detach /dev/disk4s1

We create the image using a source folder. We could also have created an empty image and then add content to it. Let's see how to do that.

1

$ hdiutil create -megabytes 10 ~/Desktop/newimage.dmg

That create s new image of ten megabytes of space. To work with that image, we need to mount it. We are going to mount it but without including its filesystem.

1
2
3
4

$ hdiutil attach -nomount ~/Desktop/newimage.dmg
hdiutil attach -nomount ~/Desktop/newimage.dmg
/dev/disk3       GUID_partition_scheme
/dev/disk3s1      Apple_HFS

 DANGER, WILL ROBINSON! PLEASE MAKE SURE YOU ARE POINTING TO THE CORRECT DEVICE, ELSE YOU CAN LOSE DATA! I WON'T BE RESPONSIBLE FOR ANYTHING YOU DO WITH THE INSTRUCTIONS I POST HERE, TREAD CAREFULLY. 

In my computer the image is mounted in /dev/disk3 it might use a different one on your computer so make sure you refer to the correct one or BAD THINGS WILL HAPPEN.

Why did we mount our image this way? Well, we did it so we can format the volume using diskutil. Let's do that.

1

$ diskutil partitionDisk disk3 GPT APFS "My New Image APFS" 100%

Now we can add some content to that Image.

1

$ echo "Hello, world!" > /Volumes/My\ New\ Image\ APFS/howdy.txt

Now we can eject the volume.

1
2
3

$ hdiutil detach /dev/disk3
# Remember you could also do
# hdiutil detach /Volumes/My\ New\ Image\ APFS

And now you can mount it again, and you'll see your howdy.txt file. The images we've created are Read/Write. We can also convert it to a read-only image:

1

$ hdiutil convert -format UDRO ~/Desktop/newimage.dmg -o ~/Desktop/ronewimage.dmg

You can see all the supported formats in the man page. Here is the list in mine:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

UDRW - UDIF read/write image
UDRO - UDIF read-only image
UDCO - UDIF ADC-compressed image
UDZO - UDIF zlib-compressed image
ULFO - UDIF lzfse-compressed image (OS X 10.11+ only)
ULMO - UDIF lzma-compressed image (macOS 10.15+ only)
UDBZ - UDIF bzip2-compressed image (Mac OS X 10.4+ only)
UDTO - DVD/CD-R master for export
UDSP - SPARSE (grows with content)
UDSB - SPARSEBUNDLE (grows with content; bundle-backed)
UFBI - UDIF entire image with MD5 checksum
UDRo - UDIF read-only (obsolete format)
UDCo - UDIF compressed (obsolete format)
RdWr - NDIF read/write image (deprecated)
Rdxx - NDIF read-only image (Disk Copy 6.3.3 format; deprecated)
ROCo - NDIF compressed image (deprecated)
Rken - NDIF compressed (obsolete format)
DC42 - Disk Copy 4.2 image (obsolete format)

Now if we mount our new image and try to write to it we'll get an error:

1
2
3
4
5

$ hdiutil attach ~/Desktop/ronewimage.dmg 
$ echo "bye" > /Volumes/My\ New\ Image\ APFS/bye.txt
-bash: /Volumes/My New Image APFS/bye.txt: Read-only file system
# Let's detach it to go back to a clear state
$ hdiutil detach /Volumes/My\ New\ Image\ APFS

Alright, that should give us enough background on Disk Images (.dmg) to build our ISO image for Catalina. Let's do that now.

Creating macOS Catalina ISO image

Since Mac OS X Lion (10.7 - released in 2011), Apple decided to no longer distribute an installation DVD, it provides the installer as an application that can be downloaded via the AppStore. This installer application contains the base system for installation and also a Recovery image that includes a basic OS implementation that is used at boot time. When you boot to the Recovery volume, you can use it to restore and/or install the operating system. You can read more on This detailed article on AFP548.

The idea is to extract all the parts from the Install macOS Catalina.app application bundle to build our ISO. I'm going to use Jeff Geerling's macOS VirtualBox VM script as a basis. It is similar to the instructions in Kedy Liu's article on macOS Kernel Debugging.

We are going to mainly focus on the InstallESD.dmg and BaseSystem.dmg images that come inside the application installer to generate an ISO image.

If you are not familiar with the ISO format, an ISO image is the image format used on an optical disk. It is the format used by DVD, CD, or Blue-ray Disc. In the past, it was common to use CDs as the media to install the operating systems. VirtualBox supports that format. We plan to create an ISO image from the installer.

Let's begin.

Extracting the BaseSystem.dmg

After the AppStore download is complete, you'll be able to find the Install macOS Catalina.app inside your /Applications/ directory.

I'll be using macOS Catalina as an example, but as long as you have the installer of any OS newer than 10.7, the process should be similar.

Navigate to the /Applications/Install macOS Catalina.app/Contents/SharedSupport/ directory, and list the files:

1
2
3
4
5
6
7

.
├── AppleDiagnostics.chunklist
├── AppleDiagnostics.dmg
├── BaseSystem.chunklist
├── BaseSystem.dmg
├── InstallESD.dmg
└── InstallInfo.plist

Excellent, we have the BaseSystem.dmg and the InstallESD.dmg. Inside the InstallESD.dmg we'll find the installer that uses the BaseSystem image to install the OS.

To extract the installer we'll need to mount the InstallESD.dmg image.

1
2
3
4

$ hdiutil attach \
/Applications/Install\ macOS\ Catalina.app/Contents/SharedSupport/InstallESD.dmg \
-noverify -nobrowse -mountpoint /Volumes/install_app
# that should mount the iamge in /Volumes/install_app

The BaseSystem.dmg has an image formatted in HFS. We can create an empty image as we did in the last example of the previous section. Then we can "restore" the BaseSystem image in the newly created image. It sounds confusing, but you'll understand it better when we begin doing it.

Let's begin by creating the destination image. I'll show you the command first and after I'll explain what it does.

1
2

$ hdiutil create -o ~/Desktop/Catalina -size 9g\
 -layout SPUD -fs HFS+J -type SPARSE

We are creating a sparse image. The sparse image type means that the image we create will only take as much space as the content we put on it, with the limit of 9 Gigabytes that we set. The layout indicates the partition table's layout for the created volume. We are stating that we want a single entry. The partition table layout can be split in multiple /dev entries. For example, we could have one for the GUID Partition Table (GPT), and the other partitions have the data. With the SPUD layout, we will only have one /dev entry. If you want to learn more visit the following links about Apple partition Map and GUID Partition Table.

We also define the file system type to be HFS+J. The BaseSystem.dmg disk image is also in HFS, so that is why we choose it (also because hdiutil can only resize HFS).

Ok, now you can run the command to create the sparse image. I'm showing you the command again:

1
2
3

$ hdiutil create -o ~/Desktop/Catalina -size 9g\
 -layout SPUD -fs HFS+J -type SPARSE
# This creates the sparse image Catalina.sparseimage in your ~/Desktop

Let's mount it so we can work on it.

1
2

$ hdiutil attach ~/Desktop/Catalina.sparseimage\
  -mountpoint /Volumes/install_build

Now we are going to copy the BaseSystem.dmg image into our /Volume/install_build image.

To copy the image we are going to use the asr command (you can check the man page asr(8) for more details).

1

$ asr restore -source /Applications/Install\ macOS\ Catalina.app/Contents/SharedSupport/BaseSystem.dmg -target /Volumes/install_build --noprompt -noverify -erase

Alright, this will copy the Base System inside our volume /Volumes/install_build, so It will erase our /Volumes/install_build and replace it with /Volumes/macOS\ Base\ System/. If you check our /Volumes/ you'll see it there.

The BaseSystem.dmg includes symbolic links to the Packages directory. When we are installing the OS with the application that symbolic link will exist. But in our case, that symbolic link points to the wrong place. You can verify this by listing the contents in /Volumes/macOS Base System/System/Installation/.

1
2
3
4

$ ls -l /Volumes/macOS\ Base\ System/System/Installation
# You'll see something similar to this:
drwxr-xr-x 14 derik staff  476B Oct 23 18:26 CDIS/
lrwxr-xr-x  1 derik staff  33B Oct 23 18:25 Packages@ -> /System/Installation/PackagesLink

What we are going to do is replace that symbolic link with the contents of the Packages directory we find in our InstallESD.dmg that we mounted in /Volumes/install_app. We can rm and then cp Or we can use the ditto(1) command.

1

$ ditto -V /Volumes/install_app/Packages /Volumes/macOS\ Base\ System/System/Installation/

We also need to copy some installer dependencies, i.e. BaseSystem.chunklist and BaseSystem.dmg, and add them to the image. We find those two files in /Applications/Install macOS Catalina.app/Contents/SharedSupport, let's list that directory:

1
2
3
4
5
6
7
8

$ tree -L 1 /Applications/Install\ macOS\ Catalina.app/Contents/SharedSupport/
/Applications/Install\ macOS\ Catalina.app/Contents/SharedSupport/
├── AppleDiagnostics.chunklist
├── AppleDiagnostics.dmg
├── BaseSystem.chunklist
├── BaseSystem.dmg
├── InstallESD.dmg
└── InstallInfo.plis

Ok, now that we know where they are let's copy them to our image volume:

1

$ ditto -V /Applications/Install\ macOS\ Catalina.app/Contents/SharedSupport/BaseSystem.{chunklist,dmg} /Volumes/macOS\ Base\ System/

That's it. That is our image ready. Let's now unmount the images, clean up and prepare to convert our installation image to a read-only image.

1
2

$ hdiutil detach /Volumes/install_app
$ hdiutil detach /Volumes/macOS\ Base\ System/

Let's clean up and resize to free up any extra space. hdiutil can only resize filesystems of type HFS+. Again, read the man page for additional information on resize (hdiutil(1)). We can obtain the current size of the image using the -limit argument:

NOTE: as pointed out by quantum_libet we can simplify by using the size -min parameter, this is much more clean than my previous solution. Thanks for the tip!

1
2
3
4

hdiutil resize -size min ~/Desktop/Catalina.sparseimage
# The following instruction is how I obtain the size
#$ hdiutil resize -limits ~/Desktop/Catalina.sparseimage | tail -n 1 | awk '{ print $1 }'
# in my case 18721976 sectors of 512bytes. That is 9.58Gbs

Now we are ready to convert the image to a CD-R export image. You can see how to do this in the man page in the examples section (hdiutil(1), as you can see a lot of information in the man page).

1
2
3

$ hdiutil convert ~/Desktop/Catalina.sparseimage -format UDTO -o ~/Desktop/Catalina
# This will generate ~/Desktop/Catalina.cdr
$ mv ~/Desktop/Catalina.cdr ~/Desktop/Catalina.iso

And that's it. We now have the Catalina.iso. We can use our ISO in VirtualBox to create our macOS Virtual machine!

Testing the ISO in VirtualBox

Alright, now for the ugly part. VirtualBox doesn't support APFS in their UEFI boot loader (you can read the discussion in VirtualBox's forum). Luckily Alexander Willner created a handy script with the name runMacOSinVirtualBox that automates the creation of a VirtualBox VM that works. The script creates a bootable partition that can be used to boot macOS while we wait for VirtualBox to support APFS.

His script can also be used to create an ISO, but we have already done that, so no need.

Ok clone or download his script and run it.

1
2
3
4
5

# I'll clone it
$ git clone https://github.com/AlexanderWillner/runMacOSinVirtualBox.git
$ cd runMacOSinVirtualBox
# I'm going to name my Virtual Machine macOS-Catalina
$ VM_NAME=macOS-Catalina make vm

Now we can go to VirtualBox, and add our ISO image as an optical drive to the newly created VM.

Go to: Settings > Storage > Add New Optical Drive and select our newly created Catalina.iso.

Now you can start the VM. Now select Disk Utility and format your macOS partition using APFS (click Erase and select APF). After the formatting is complete, quit Disk Utility and click "Reinstall macOS". Follow and complete the installation process.

Congratulations! You now have Catalina running on VirtualBox :).

Final thoughts

We could have copy/paste or cloned the scripts created by Alexander Willner, Jeff Greeling or Kedy Liu. But I believe it's helpful to understand any code we use so we can build on top of that knowledge.

We learned how to work with Disk Images. As you can see, it is a convenient topic to understand. For example, some people distribute their applications outside the AppStore using .dmgs.

And also you now have a VM that you can use to build your security lab or to explore Kernel Debugging.

Ok, that's it for this post. I hope that you found this post useful :).

I also would like to thank all the people that very openly share their knowledge and help us understand how things work. I think the metaphor standing on the shoulders of giants, is quite apt.

Thank you.

Related topics/notes of interest

Most of the links I'll add here are already in the post but for convenience I'll add them again.


** If you want to check what else I'm currently doing, be sure to follow me on twitter @rderik or subscribe to the newsletter. If you want to send me a direct message, you can send it to derik@rderik.com.