Developing a non-x86 Operating system

Developing an (RT)OS is not a trivial task. It is very educational though. My advice to you is to start hardware independent. PC is a good starting point as it comes with plenty of I/O possibilities and good debugging. If you create a kind-of-virtual machine application, you can create something with simple platform capabilities (console output, some buttons/indicators are a good start). Also, you can use files for instance, to output timing (schedules) If you start on 'bare metal' you'll have to start from scratch. Debugging on a LED (on/off/blinking) is very hard and time consuming. My second advice is to define your scope early: is it the scheduler, the communication mechanisms or the file systems you're interested at... ? Doing all can easily end up in a life long project.

Samek, Miro, Practical UML Statecharts in C/C++ contains some interesting sections on a microkernel. It's one of my favorite books. Advanced PIC Microcontroller Projects in C: From USB to RTOS with the PIC 18F Series seems to cover some of your interests; I haven't read it yet though. Operating Systems: Internals and Design Principles may also bring good insights. It covers all aspects from scheduler to network stack. Good luck!

Seems like you should get a copy of Jean Labrosse's book MicroC/OS.

It looks like he may have just updated it too.

http://micrium.com/page/press_room/news/id:40

http://micrium.com/page/home

This is a well documented book describing the inner workings of an RTOS written in C and ported to many embedded processors. You could also run it on a x86, and then cross compile to another processor.

Contiki might be a good thing to research. It's very small, runs on microcontrollers, and is open source. It has a heavy bias towards networking and communications, but perhaps you can skip those parts and focus on the kernel.