Solving overdetermined system by QR decomposition
$begingroup$
I need to solve $Ax=b$ in lots of ways using QR decomposition.
$$A = begin{bmatrix}
1 & 1 \
-1 & 1 \
1 & 2
end{bmatrix}, b = begin{bmatrix}
1 \
0 \
1
end{bmatrix}$$
This is an overdetermined system. That is, it has more equations than needed for a unique solution.
I need to find $min ||Ax-b||$. How should I solve it using QR?
I know that QR can be used to reduce the problem to
$$Vert Ax - b Vert = Vert QRx - b Vert = Vert Rx - Q^{-1}b Vert.$$
but what do I do after this?
linear-algebra numerical-methods numerical-linear-algebra
$endgroup$
add a comment |
$begingroup$
I need to solve $Ax=b$ in lots of ways using QR decomposition.
$$A = begin{bmatrix}
1 & 1 \
-1 & 1 \
1 & 2
end{bmatrix}, b = begin{bmatrix}
1 \
0 \
1
end{bmatrix}$$
This is an overdetermined system. That is, it has more equations than needed for a unique solution.
I need to find $min ||Ax-b||$. How should I solve it using QR?
I know that QR can be used to reduce the problem to
$$Vert Ax - b Vert = Vert QRx - b Vert = Vert Rx - Q^{-1}b Vert.$$
but what do I do after this?
linear-algebra numerical-methods numerical-linear-algebra
$endgroup$
add a comment |
$begingroup$
I need to solve $Ax=b$ in lots of ways using QR decomposition.
$$A = begin{bmatrix}
1 & 1 \
-1 & 1 \
1 & 2
end{bmatrix}, b = begin{bmatrix}
1 \
0 \
1
end{bmatrix}$$
This is an overdetermined system. That is, it has more equations than needed for a unique solution.
I need to find $min ||Ax-b||$. How should I solve it using QR?
I know that QR can be used to reduce the problem to
$$Vert Ax - b Vert = Vert QRx - b Vert = Vert Rx - Q^{-1}b Vert.$$
but what do I do after this?
linear-algebra numerical-methods numerical-linear-algebra
$endgroup$
I need to solve $Ax=b$ in lots of ways using QR decomposition.
$$A = begin{bmatrix}
1 & 1 \
-1 & 1 \
1 & 2
end{bmatrix}, b = begin{bmatrix}
1 \
0 \
1
end{bmatrix}$$
This is an overdetermined system. That is, it has more equations than needed for a unique solution.
I need to find $min ||Ax-b||$. How should I solve it using QR?
I know that QR can be used to reduce the problem to
$$Vert Ax - b Vert = Vert QRx - b Vert = Vert Rx - Q^{-1}b Vert.$$
but what do I do after this?
linear-algebra numerical-methods numerical-linear-algebra
linear-algebra numerical-methods numerical-linear-algebra
asked yesterday
Guerlando OCsGuerlando OCs
10821856
10821856
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
$begingroup$
The most straightforward way I know is to pass through the normal equations:
$$A^T A x = A^T b$$
and substitute in the $QR$ decomposition of $A$ (with the convention $Q in mathbb{R}^{m times n},R in mathbb{R}^{n times n}$). Thus you get
$$R^T Q^T Q R x = R^T Q^T b.$$
But $Q^T Q=I_n$. (Note that in this convention $Q$ isn't an orthogonal matrix, so $Q Q^T neq I_m$, but this doesn't matter here.) Thus:
$$R^T R x = R^T Q^T b.$$
If $A$ has linearly independent columns (as is usually the case with overdetermined systems), then $R^T$ is injective, so by multiplying both sides by the left inverse of $R^T$ you get
$$Rx=Q^T b.$$
This system is now easy to solve numerically.
For numerical purposes it's important that the removal of $Q^T Q$ and $R^T$ from the problem is done analytically, and in particular $A^T A$ is never constructed numerically.
$endgroup$
1
$begingroup$
Is there any reason to make this so convoluted? From $Ax=b$ you have $QRx=b$, multiply by $Q^T$ on the left.
$endgroup$
– Martin Argerami
yesterday
$begingroup$
@MartinArgerami Because actually the least squares solution usually does not satisfy $Ax=b$. This simple perspective only shows you that this approach gives you a solution when a solution exists. Now you could argue directly that multiplying both sides by $Q^T$ furnishes an equation whose solution is the least squares solution. (Such an argument would resemble the usual geometric argument for deriving the normal equations.) This would make a good alternative answer to mine.
$endgroup$
– Ian
yesterday
add a comment |
$begingroup$
Note that $Rx$ has the form
$$Rx = begin{bmatrix} y_1 \ y_2 \ 0end{bmatrix} $$
, so if $$ Q^{-1}b = begin{bmatrix} z_1 \ z_2 \ z_3end{bmatrix}$$
then $|| Rx - Q^{-1}b||$ will be minimal for $y_1 = z_1$, $y_2=z_2$. This set of equation is no longer overdetermined.
Using matrix notation, if tou write $R = begin{bmatrix} R_1 \ 0end{bmatrix}$ and intoduce $P=begin{bmatrix}1 & 0 & 0 \ 0 & 1& 0end{bmatrix}$, then you have
$$ R_1x = PQ^{-1}b$$
$$ x = (R_1)^{-1}PQ^{-1}b$$
$endgroup$
1
$begingroup$
The key trick in this answer is that by the orthogonality, $| Ax - b | = | Rx - Q^T b |$.
$endgroup$
– Ian
yesterday
$begingroup$
@Ian, That's something that OP has alredy obtained on his own (since $Q$ is orthogonal, $Q^{-1}=Q^T$).
$endgroup$
– Adam Latosiński
14 hours ago
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "69"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3185239%2fsolving-overdetermined-system-by-qr-decomposition%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
The most straightforward way I know is to pass through the normal equations:
$$A^T A x = A^T b$$
and substitute in the $QR$ decomposition of $A$ (with the convention $Q in mathbb{R}^{m times n},R in mathbb{R}^{n times n}$). Thus you get
$$R^T Q^T Q R x = R^T Q^T b.$$
But $Q^T Q=I_n$. (Note that in this convention $Q$ isn't an orthogonal matrix, so $Q Q^T neq I_m$, but this doesn't matter here.) Thus:
$$R^T R x = R^T Q^T b.$$
If $A$ has linearly independent columns (as is usually the case with overdetermined systems), then $R^T$ is injective, so by multiplying both sides by the left inverse of $R^T$ you get
$$Rx=Q^T b.$$
This system is now easy to solve numerically.
For numerical purposes it's important that the removal of $Q^T Q$ and $R^T$ from the problem is done analytically, and in particular $A^T A$ is never constructed numerically.
$endgroup$
1
$begingroup$
Is there any reason to make this so convoluted? From $Ax=b$ you have $QRx=b$, multiply by $Q^T$ on the left.
$endgroup$
– Martin Argerami
yesterday
$begingroup$
@MartinArgerami Because actually the least squares solution usually does not satisfy $Ax=b$. This simple perspective only shows you that this approach gives you a solution when a solution exists. Now you could argue directly that multiplying both sides by $Q^T$ furnishes an equation whose solution is the least squares solution. (Such an argument would resemble the usual geometric argument for deriving the normal equations.) This would make a good alternative answer to mine.
$endgroup$
– Ian
yesterday
add a comment |
$begingroup$
The most straightforward way I know is to pass through the normal equations:
$$A^T A x = A^T b$$
and substitute in the $QR$ decomposition of $A$ (with the convention $Q in mathbb{R}^{m times n},R in mathbb{R}^{n times n}$). Thus you get
$$R^T Q^T Q R x = R^T Q^T b.$$
But $Q^T Q=I_n$. (Note that in this convention $Q$ isn't an orthogonal matrix, so $Q Q^T neq I_m$, but this doesn't matter here.) Thus:
$$R^T R x = R^T Q^T b.$$
If $A$ has linearly independent columns (as is usually the case with overdetermined systems), then $R^T$ is injective, so by multiplying both sides by the left inverse of $R^T$ you get
$$Rx=Q^T b.$$
This system is now easy to solve numerically.
For numerical purposes it's important that the removal of $Q^T Q$ and $R^T$ from the problem is done analytically, and in particular $A^T A$ is never constructed numerically.
$endgroup$
1
$begingroup$
Is there any reason to make this so convoluted? From $Ax=b$ you have $QRx=b$, multiply by $Q^T$ on the left.
$endgroup$
– Martin Argerami
yesterday
$begingroup$
@MartinArgerami Because actually the least squares solution usually does not satisfy $Ax=b$. This simple perspective only shows you that this approach gives you a solution when a solution exists. Now you could argue directly that multiplying both sides by $Q^T$ furnishes an equation whose solution is the least squares solution. (Such an argument would resemble the usual geometric argument for deriving the normal equations.) This would make a good alternative answer to mine.
$endgroup$
– Ian
yesterday
add a comment |
$begingroup$
The most straightforward way I know is to pass through the normal equations:
$$A^T A x = A^T b$$
and substitute in the $QR$ decomposition of $A$ (with the convention $Q in mathbb{R}^{m times n},R in mathbb{R}^{n times n}$). Thus you get
$$R^T Q^T Q R x = R^T Q^T b.$$
But $Q^T Q=I_n$. (Note that in this convention $Q$ isn't an orthogonal matrix, so $Q Q^T neq I_m$, but this doesn't matter here.) Thus:
$$R^T R x = R^T Q^T b.$$
If $A$ has linearly independent columns (as is usually the case with overdetermined systems), then $R^T$ is injective, so by multiplying both sides by the left inverse of $R^T$ you get
$$Rx=Q^T b.$$
This system is now easy to solve numerically.
For numerical purposes it's important that the removal of $Q^T Q$ and $R^T$ from the problem is done analytically, and in particular $A^T A$ is never constructed numerically.
$endgroup$
The most straightforward way I know is to pass through the normal equations:
$$A^T A x = A^T b$$
and substitute in the $QR$ decomposition of $A$ (with the convention $Q in mathbb{R}^{m times n},R in mathbb{R}^{n times n}$). Thus you get
$$R^T Q^T Q R x = R^T Q^T b.$$
But $Q^T Q=I_n$. (Note that in this convention $Q$ isn't an orthogonal matrix, so $Q Q^T neq I_m$, but this doesn't matter here.) Thus:
$$R^T R x = R^T Q^T b.$$
If $A$ has linearly independent columns (as is usually the case with overdetermined systems), then $R^T$ is injective, so by multiplying both sides by the left inverse of $R^T$ you get
$$Rx=Q^T b.$$
This system is now easy to solve numerically.
For numerical purposes it's important that the removal of $Q^T Q$ and $R^T$ from the problem is done analytically, and in particular $A^T A$ is never constructed numerically.
edited yesterday
answered yesterday
IanIan
69.2k25392
69.2k25392
1
$begingroup$
Is there any reason to make this so convoluted? From $Ax=b$ you have $QRx=b$, multiply by $Q^T$ on the left.
$endgroup$
– Martin Argerami
yesterday
$begingroup$
@MartinArgerami Because actually the least squares solution usually does not satisfy $Ax=b$. This simple perspective only shows you that this approach gives you a solution when a solution exists. Now you could argue directly that multiplying both sides by $Q^T$ furnishes an equation whose solution is the least squares solution. (Such an argument would resemble the usual geometric argument for deriving the normal equations.) This would make a good alternative answer to mine.
$endgroup$
– Ian
yesterday
add a comment |
1
$begingroup$
Is there any reason to make this so convoluted? From $Ax=b$ you have $QRx=b$, multiply by $Q^T$ on the left.
$endgroup$
– Martin Argerami
yesterday
$begingroup$
@MartinArgerami Because actually the least squares solution usually does not satisfy $Ax=b$. This simple perspective only shows you that this approach gives you a solution when a solution exists. Now you could argue directly that multiplying both sides by $Q^T$ furnishes an equation whose solution is the least squares solution. (Such an argument would resemble the usual geometric argument for deriving the normal equations.) This would make a good alternative answer to mine.
$endgroup$
– Ian
yesterday
1
1
$begingroup$
Is there any reason to make this so convoluted? From $Ax=b$ you have $QRx=b$, multiply by $Q^T$ on the left.
$endgroup$
– Martin Argerami
yesterday
$begingroup$
Is there any reason to make this so convoluted? From $Ax=b$ you have $QRx=b$, multiply by $Q^T$ on the left.
$endgroup$
– Martin Argerami
yesterday
$begingroup$
@MartinArgerami Because actually the least squares solution usually does not satisfy $Ax=b$. This simple perspective only shows you that this approach gives you a solution when a solution exists. Now you could argue directly that multiplying both sides by $Q^T$ furnishes an equation whose solution is the least squares solution. (Such an argument would resemble the usual geometric argument for deriving the normal equations.) This would make a good alternative answer to mine.
$endgroup$
– Ian
yesterday
$begingroup$
@MartinArgerami Because actually the least squares solution usually does not satisfy $Ax=b$. This simple perspective only shows you that this approach gives you a solution when a solution exists. Now you could argue directly that multiplying both sides by $Q^T$ furnishes an equation whose solution is the least squares solution. (Such an argument would resemble the usual geometric argument for deriving the normal equations.) This would make a good alternative answer to mine.
$endgroup$
– Ian
yesterday
add a comment |
$begingroup$
Note that $Rx$ has the form
$$Rx = begin{bmatrix} y_1 \ y_2 \ 0end{bmatrix} $$
, so if $$ Q^{-1}b = begin{bmatrix} z_1 \ z_2 \ z_3end{bmatrix}$$
then $|| Rx - Q^{-1}b||$ will be minimal for $y_1 = z_1$, $y_2=z_2$. This set of equation is no longer overdetermined.
Using matrix notation, if tou write $R = begin{bmatrix} R_1 \ 0end{bmatrix}$ and intoduce $P=begin{bmatrix}1 & 0 & 0 \ 0 & 1& 0end{bmatrix}$, then you have
$$ R_1x = PQ^{-1}b$$
$$ x = (R_1)^{-1}PQ^{-1}b$$
$endgroup$
1
$begingroup$
The key trick in this answer is that by the orthogonality, $| Ax - b | = | Rx - Q^T b |$.
$endgroup$
– Ian
yesterday
$begingroup$
@Ian, That's something that OP has alredy obtained on his own (since $Q$ is orthogonal, $Q^{-1}=Q^T$).
$endgroup$
– Adam Latosiński
14 hours ago
add a comment |
$begingroup$
Note that $Rx$ has the form
$$Rx = begin{bmatrix} y_1 \ y_2 \ 0end{bmatrix} $$
, so if $$ Q^{-1}b = begin{bmatrix} z_1 \ z_2 \ z_3end{bmatrix}$$
then $|| Rx - Q^{-1}b||$ will be minimal for $y_1 = z_1$, $y_2=z_2$. This set of equation is no longer overdetermined.
Using matrix notation, if tou write $R = begin{bmatrix} R_1 \ 0end{bmatrix}$ and intoduce $P=begin{bmatrix}1 & 0 & 0 \ 0 & 1& 0end{bmatrix}$, then you have
$$ R_1x = PQ^{-1}b$$
$$ x = (R_1)^{-1}PQ^{-1}b$$
$endgroup$
1
$begingroup$
The key trick in this answer is that by the orthogonality, $| Ax - b | = | Rx - Q^T b |$.
$endgroup$
– Ian
yesterday
$begingroup$
@Ian, That's something that OP has alredy obtained on his own (since $Q$ is orthogonal, $Q^{-1}=Q^T$).
$endgroup$
– Adam Latosiński
14 hours ago
add a comment |
$begingroup$
Note that $Rx$ has the form
$$Rx = begin{bmatrix} y_1 \ y_2 \ 0end{bmatrix} $$
, so if $$ Q^{-1}b = begin{bmatrix} z_1 \ z_2 \ z_3end{bmatrix}$$
then $|| Rx - Q^{-1}b||$ will be minimal for $y_1 = z_1$, $y_2=z_2$. This set of equation is no longer overdetermined.
Using matrix notation, if tou write $R = begin{bmatrix} R_1 \ 0end{bmatrix}$ and intoduce $P=begin{bmatrix}1 & 0 & 0 \ 0 & 1& 0end{bmatrix}$, then you have
$$ R_1x = PQ^{-1}b$$
$$ x = (R_1)^{-1}PQ^{-1}b$$
$endgroup$
Note that $Rx$ has the form
$$Rx = begin{bmatrix} y_1 \ y_2 \ 0end{bmatrix} $$
, so if $$ Q^{-1}b = begin{bmatrix} z_1 \ z_2 \ z_3end{bmatrix}$$
then $|| Rx - Q^{-1}b||$ will be minimal for $y_1 = z_1$, $y_2=z_2$. This set of equation is no longer overdetermined.
Using matrix notation, if tou write $R = begin{bmatrix} R_1 \ 0end{bmatrix}$ and intoduce $P=begin{bmatrix}1 & 0 & 0 \ 0 & 1& 0end{bmatrix}$, then you have
$$ R_1x = PQ^{-1}b$$
$$ x = (R_1)^{-1}PQ^{-1}b$$
answered yesterday
Adam LatosińskiAdam Latosiński
6068
6068
1
$begingroup$
The key trick in this answer is that by the orthogonality, $| Ax - b | = | Rx - Q^T b |$.
$endgroup$
– Ian
yesterday
$begingroup$
@Ian, That's something that OP has alredy obtained on his own (since $Q$ is orthogonal, $Q^{-1}=Q^T$).
$endgroup$
– Adam Latosiński
14 hours ago
add a comment |
1
$begingroup$
The key trick in this answer is that by the orthogonality, $| Ax - b | = | Rx - Q^T b |$.
$endgroup$
– Ian
yesterday
$begingroup$
@Ian, That's something that OP has alredy obtained on his own (since $Q$ is orthogonal, $Q^{-1}=Q^T$).
$endgroup$
– Adam Latosiński
14 hours ago
1
1
$begingroup$
The key trick in this answer is that by the orthogonality, $| Ax - b | = | Rx - Q^T b |$.
$endgroup$
– Ian
yesterday
$begingroup$
The key trick in this answer is that by the orthogonality, $| Ax - b | = | Rx - Q^T b |$.
$endgroup$
– Ian
yesterday
$begingroup$
@Ian, That's something that OP has alredy obtained on his own (since $Q$ is orthogonal, $Q^{-1}=Q^T$).
$endgroup$
– Adam Latosiński
14 hours ago
$begingroup$
@Ian, That's something that OP has alredy obtained on his own (since $Q$ is orthogonal, $Q^{-1}=Q^T$).
$endgroup$
– Adam Latosiński
14 hours ago
add a comment |
Thanks for contributing an answer to Mathematics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3185239%2fsolving-overdetermined-system-by-qr-decomposition%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown