Thread by @sortiecat

Thread: After 6 years of #osdev experience, here are my recommendations for trying out making your own Operating System from scratch:

The operating systems development (osdev) wiki is awesome and has the tutorial you should start with: wiki.osdev.org/Bare_Bones

If I still have your attention, you need to read the rest of this thread before starting. You need to read all the instructions carefully!

There's a lot of information on the wiki and almost all of it exists for a reason. Read it carefully three times before asking for help.

Don't follow tutorials if they 1) can't be community edited, 2) were written by the inexperienced, 3) or haven't been peer reviewed!

There's loads of misinformation and bad advice out there! Stick to the osdev wiki, if imperfect!

There's a document with introductory words you should read: wiki.osdev.org/Getting_Started

There's a document with key things you probably should know: wiki.osdev.org/Required_Knowl…

You don't need to have all the "Required" knowledge. Some knowledge can make up for lack of other knowledge! Learn as you go along!

Operating systems development is a many year process. Have patience, have fun, know why you are here. Fun and learning are good reasons!

Here's a list of common failure modes, don't do this at home: wiki.osdev.org/Beginner_Mista…

If you're interested in what amazing hobby operating systems people have built, see the list: wiki.osdev.org/Notable_Projec…

My hobbyist system operating system is Sortix. It's self-hosting now after 6 years. Ports of lots of cool stuff.

If you want to make an operating system similar to what I've done, here are some tips:

Keep going at it! You probably won't succeed at first, but you'll learn something, and be better prepared when you return months/years later

The osdev forum is a place where you can ask for help in a forum setting: forum.osdev.org - I'm a moderator there.

The #osdev IRC channel on the Freenode network is great for real time access to experts around the world. I love the place.

IRC is almost real time. Don't ask a question and leave 30 seconds later. I'm upstairs making coffee. Or at work.

If you follow an tutorial outside the osdev wiki, ask in #osdev whether it has a known list of defects. It probably has!

Certainly do not follow Youtube tutorials. 1) Not editable 2) Made by inexperienced 3) Not peer reviewed.

Some of the very best looking tutorials have subtle very nasty bugs that hit you months/years from now. Calling conventions can be tricky.

You should write a 32-bit kernel for x86 CPUs. Don't write a bootloader. Don't start with 64-bit as the boot is slightly more complicated.

Bootloaders teach you a lot of complexity about how CPUS worked many years ago. It's largely irrelevant details. The A20 line is not fun.

Writing an operating system in assembly might be fun. I don't recommend it. There's a reason why programming languages were invented.

Some assembly OS projects have gotten terrifyingly far. I don't really consider them to have a real future. Always try for a real future.

It's tempting to something "simple that works, skipping corners". I get that, I did it. But there's very good lessons in doing real systems.

It's hell paying back the debt afterwards, to become a real system. I'm still paying off 2011, 2012, 2013, 2014, 2015, 2016, 2017 debt.

The kernel is where interesting stuff happens, and where you start to make a real design. These are the fun parts I recommend!

Making an user-space is also great when you get that, designing a good system call layer is great, making an awesome user-space.

If you want to port user-space software, at some point your project becomes a "Linux From Scratch", or distribution-level stuff. Fun!

It doesn't really matter if your kernel is 32-bit or 64-bit. It's largely the same thanks to an actual programming language.

You only need a few bits of assembly here and there to get you don't. Be wary of inline assembly, prefer actual assembly files.

Inline assembly is terrifyingly difficult to get correct, even if it appears to work. Always consult the GCC docs and #osdev experts.

It's always safe to do an assembly file instead of inline assembly. Do if in doubt.

Osdev is full of crazy people. You're one of them and probably don't even realize it. Welcome!

Paging is called paging because you can't possibly expect to figure it out without drawing your paging tables on paper.

Don't do a 16-bit real mode operating system. Might be fun to learn some details, but your end result will be irrelevant.

Try to do a real operating system. Think about security from the get go. Multithreading. Do the best you can and iterate.

Yes, absolutely, your thing right now is irrelevant. Mine was 6 years ago. Now? I'm really excited about what it might grow into over time.

Don't prevent your small operating system from becoming something real, even if you don't expect it to.

Don't use flat binaries: They are too simple and contain no information where they are loaded. Use ELF instead.

Statically linked ELF programs are surprisingly simple and easy to load. Go straight for using that for your kernel and user-mode programs.

Curiously flat binaries are surprisingly hard to get right and are poorly supported. You'll actually save time using ELF for your programs.

Don't listen to your own advice. You are a newbie and might not know it. You'll hate your own code deeply a few years down the road.

Don't try to replace C or Unix without truly knowing what's wrong with it and what's good about it. Lots of stuff to improve or learn.

Implementing POSIX has value. You'll be an Unix expert by the end. Very good if you want a great job.

If your did your own very custom design, maybe that is all you learned? Think it through and make a good choice.

If you implement a POSIX system, what libc? I did my own. Lots of learning and room for excellence. It takes more time.

I recommend musl for your libc. You should implement the Linux syscalls. Only a small part of a kernel, plenty of room to customize.

If you want to diverge, you can always just patch musl. It was ported to Windows through midipix!

Don't use newlib. It seems simple, but it's too simple, which means it's too complex. It's low quality for embedded. musl is for real system

Don't do black magic. Do it the normal way, the complexity may be necessary, therefore it is simple and think nothing special of it.

Don't use code you developed while sleeping. You were unconscious. Do you code well when you are really tired?

I once dreamed a three assembly instruction solution to a problem, but when I woke up and coded it, it took six instructions.

Don't leave proving the code is correct as an exercise for the reader.

Don't proclaim me a master of osdev and quote my holy words in #osdev. (It happened.)

I'm really just an inexperienced guy that put in a bunch of work for 6 years. Remember that about gaps in knowledge? I have plenty still.

Understand what your goals truly are and what you want from osdev.

Maybe what you really want to do is a desktop environment or a language runtime.

Don't play the role of an operating systems developer, *be* one.

With every incompatibility come a price that must be paid.

Corollary: Pick your battles.

Read instructions carefully.

Take correctness seriously.

Correct code is secure.

Know your programming language well.

Know your ABI well.

Be truly afraid of undefined behavior.

No really, learn what undefined behavior. is It's the enemy of correctless and security. Avoid it at all costs. No black magic! Do it normal

Use a modern, good Unix for osdev.

Sorry, macOS is not that. You can still use it, but a lot of tools Just Work® Better™ on a modern Unix. I recommend Linux or the great BSDs.

Use UTF-8 (no BOM, Unix line terminators).

Your operating system should be itself, not fight what it is.

^ That is a key design principle of mine. Do it the normal way. No multiple personalities or compatibility. Just normal simplicity.

What other operating systems do: Embrace it, or fight it. Don't half-adopt it.

*Never lie to your compiler*. It's undefined behavior. See above, you've learned to be terrified of this.

Use a cross-compiler: wiki.osdev.org/GCC_Cross-Comp… - Your OS is not Linux. Never lie to your compiler! It *will backfire badly months from now*!

clang comes with cross-compiles depending on your distro, but "clang is native a cross-compiler" is nonsense. You need -T i686-elf or such.

I still very much recommend osdev'ing with gcc rather than clang. It's much better understood by the community. We have guides for porting.

Here's a guide for teaching gcc how to target your operating system: wiki.osdev.org/OS_Specific_To… - Don't do this until you make an user-space.

The Bare Bones tutorial -- you'll read this after this thread -- is just that. It's the bare minimum to get a hello world going.

I have made an template hello world operating system that organizes itself more like a real one would: wiki.osdev.org/Meaty_Skeleton

It cross-compiles things properly, component by component, installing into the sysroot, a subdirectory that is your OS's future root dir

Initialize at compile, link or load time, not run-time.

Early environments are bad! They're special times where the rules are different. Lots of room for bugs. Minimize their existence!

Have a pre-emptive kernel early on.

I recommend a fully preemptive multithreaded. System calls should be preemptive. To block they just don't return immediately.

Don't do big locks or cooperative scheduling. It's not as simple, harder to get right, less efficient.

Every thread should have its own stack to process system calls on.

1:1 threading is the right way. User-space scheduling is crazy. I don't distinct between user and kernel threads, same thing.

I recommend ext2 as your first filesystem. It's simple and has all the basic features of a real filesystem. It'll make you do stuff well.

Certainly do not use FAT for your first filesystem. It's limitations will probably just give your VFS brain damage unless you did a real FS.

Be an expert.

Osdev is really all about becoming an expert. You don't have to start out as one, but you need to become one in all you do.

Osdev will have amazing and terrifying bugs. You don't need to know everything, but you what you do, you need to know it.

You need a good mental state of the machine and your code. If debugging C fails, use objdump -d to check the instructions are right.

Just release your hobbyist OS source code already.

Your code is very probably terrible - be sure your README tells people not to learn from you (you will not be an expert for several years).

But don't natively protect your precious code. It's not. If you publish it with git, you can link it, and people can help.

If you publish your code with git, people like me might actually read it and offer a lot of useful advice. “Don't do that, etc.”.

I can tell a lot about a person from the headers.

Here's a list of books. Bunch of good stuff here: wiki.osdev.org/Books

It's totally possible to port software to your OS if you implement POSIX. Cool stuff! Here's a guide: wiki.osdev.org/Cross-Porting_…

Here's some thoughts on the kind of build system I like, a hard one: wiki.osdev.org/Hard_Build_Sys…

Convenient scripts that do everything for you, maybe interactively, or you source something. They are too simple and inflexible.

It's worth thinking about what sort of osdev archetype you are: wiki.osdev.org/What_order_sho…

But honestly, that is just about initial priorities. After a while, you realize you're many of them.

I once began a roadmap of how to make an operating system, it's incomplete, but possibly useful: wiki.osdev.org/Creating_an_Op…

You should think about security early on. Implement stuff like the stack protector: wiki.osdev.org/Stack_Smashing…

Implementing things like Thread Local Storage is also cool: wiki.osdev.org/Thread_Local_S…

It's a good idea with a kernel debugger and dumping useful information to the serial line.

You can attach gdb as a debugger, actually, with some tricks. But it starts to break down with threads and user-space.

Here's some thoughts on making your own libc: wiki.osdev.org/Creating_a_C_L…

You should target real hardware and make sure your OS works well on real hardware.

At the same time, let's not kid ourselves. Most people will be running your OS in a virtual machine: Target them and make them work well.

For a first "graphics driver", the Bochs Video After exists in Bochs, Qemu (with -vga std), and VirtualBox (by default). It's very easy.

OK, that should be a big iteration of lots of thoughts. Be sure to ask if you have any questions.

One day you can become self-hosting on real hardware and use your network stack to publish your builds and tweet about it natively.

I'm literally planning on doing that for my upcoming 1.1 release of my sortix.org that will hopefully happen late this year.

After that the plan is to somehow get my OS into space. Anybody knows any astronauts? “This OS came from outer space!”