How to interpet jpeg binray files?

I would like to write a program which main purpose will be to modify jpeg files (I'm using C++). But I have no idea how to interpret the jpeg binary files. My original expectation was to convert this file to ascii, and hopefully find a data structure similar to a matrix of pixels (maybe one matrix for each RGB color), make the changes directly to this matrix, and then rewrite the binary file.

Please notice that I have no idea if the process described above is feasible.

Can you give me some advice on how can I learn to interpret the binary file? Is the process that I'm suggesting plausible?

I'm also aware that there may be existing libraries capable of doing what I want, but my main purpose right now is to learn something new.