Fixed Point: Scaling Factors, Resolution and Rounding Error
This lesson is part of the course
In this lesson
You find the resolution of a sensor that sends sixteenths of a degree, store six temperatures as integers, decode them again and measure the rounding error each one picks up.
Buy the course once for $9.99 to unlock every lesson and lab, the final exam and the certificate – lifetime access, no subscription.
Lessons in this course
- Positional Notation: Decimal to Binary by Repeated Division and Back Locked
- Hexadecimal & Octal: Nibbles, Hex Digits and Colour Codes Locked
- Bit Flags & Masks: Reading, Setting and Clearing Status Bits Locked
- Two's Complement: Signed Ranges, Negative Numbers and Decoding Bytes Locked
- Integer Limits & Overflow: Max Values, Bits Needed and Counters that Roll Over Locked
- Fixed Point: Scaling Factors, Resolution and Rounding Error Locked
- Floating Point with a Toy 8-Bit Format: Sign, Exponent, Mantissa and Why 0.1 Is Not Exact Locked
- Storage Units & Characters: GB vs GiB, File Sizes and Text in Bytes Locked