When Answering This Question - I've Plugged Into M

 JonnyMathFrom: USPosts: 40Votes: 120 When answering this question - I've plugged into my calculator (ti 89) and ms excel the same numbers that you have displayed in the solutions... instead of getting for part 2 step 2 or  (ii) 0.263158 and for part (iii) 0.251256 I get 0.263160 and 0.261300.   My calculator display digits mode is set to float 7... why the discrepancy and how do I fix this?

 AnonymousFrom: -Posts: -Votes: - Ah, I was thinking of the wrong question. Nevermind; whatI was thinking of wouldn't be an issue here. I can't helpyou. Sorry!

 AnonymousFrom: -Posts: -Votes: - If it is off by only a small amount, it could be a roundingerror. On computers (and likely on the 89, though I'm unsurehere), a decimal is most often represented by the sum a_i / 2^i fori = 1 to some finite number and a_i = 1 or 0. As a result,since you are only summing powers of 2, it is impossible to get anexact solution for certain numbers (even if you have millions ofdigits). This leads to small errors. As a result, youshould never trust a computer's floating point math to more than afew digits unless the program is specifically designed forhigh-precision math (excel is not one of these programs).

 MisterMasterFrom: USPosts: 31Votes: 93 Under "Mode" if you push F2 to get page 2, do you have "exact"under the exact/approx option?

 AnonymousFrom: -Posts: -Votes: - How is your problem scaled? Computers have a particularlyhard time with precision in matrix inversion if the numbers in somerows are several orders of magnitude more than on another. You can divide all the numbers on the same row by 100 or 1000 orso.

 AnonymousFrom: -Posts: -Votes: - I was saying excel might have given you the wrong answer. Ithas been known to do so.

 AnonymousFrom: -Posts: -Votes: - I had automatic - it doesn't matter what it is on this setting - I've tried it multiple ways including using exact...   any suggestions?