* Critique of the Xen Security Process
@ 2015-11-06 17:22 Joanna Rutkowska
2015-11-06 19:41 ` James Bulpin
` (6 more replies)
0 siblings, 7 replies; 25+ messages in thread
From: Joanna Rutkowska @ 2015-11-06 17:22 UTC (permalink / raw
To: xen-devel; +Cc: qubes-devel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hello,
Recently Xen has released the XSA-148 advisory [1] addressing a fatal bug in the
hypervisor. The bug has been lurking there for the last 7 years! We, the Qubes
OS Project, have commented on this in our Security Bulletin #22 [2]. And far
from enthusiastic commentary that was (FWIW, it was me who wrote this QSB, as
evidenced in the commits log, in case some from the Xen community would like to
direct their rage towards a particular human being ;) Ian Jackson then wrote a
response on the Xen blog [3]. I was then asked to share some more thoughts about
how I thought Xen could actually improve its security process [4]. So, I share
some these below:
1. First of all, I wish Xen was somehow more defensively coded. To provide some
examples:
a. In XSA-109 [5] there was a problem with the hypervisor dereferencing a NULL
pointer. The problem was fixed by the Xen Security Team by applying a patch
which (hopefully) made sure the execution path that lead to this NULL pointer
dereferencing code was never taken. Back then I suggested (on the Xen
pre-disclosure list) to make this patch more explicit though:
> On Wed, Jan 21, 2015 at 02:31:51PM +0100, Joanna Rutkowska wrote:
> (...)
>>
>> Wouldn't it be prudent to also check if:
>>
>> (v->arch.paging.mode>{write_guest_entry,cmpxchg_guest_entry} != NULL)
>>
>> ... in the two affected functions, just before derefing these function
>> pointers?
>>
>> Going even a step further: how about replacing all
>> function-pointer-based calls with macros that first validates the
>> pointer before derefing it? At least when the system doesn't have SMEP?
>>
...to which I got a reply from one of the Xen Security Team engineers that the
above might perhaps be justified in debug builds only, followed by a standard:
"feel free to contribute a patch".
b. The XSA-123 [6] was another critical security bug in Xen, this time resulting
from one of the hypervisor developer's fetish to use an absolutely confusing
construct in order to save a few modest bytes in a structure which might have
been allocated by the system maybe a few tens of times at best. Even more
worrying was the way how Xen Security Team decided to fix the bug: again by
modifying some condition in the code further up the execution path, with the
hopes that this time they would ensure this puzzling construct would always be
used properly. We wrote more about this in our QSB #18 [7].
c. Finally, the way how Xen fixed the recent XSA-148 looks also very reactive,
IMHO. With a bug of this calibre, I would expect Xen to carefully review and
augment all its PV memory virtualization code with additional checks (ASSERTs),
ensuring certain invariants are always satisfied. Such as e.g. that none of the
pages containing PDEs or PTEs are becoming writeable by the VM.
I can't help but have a feeling that some of the Xen developers seem to be
overconfident in their belief they can fully understand all the possible
execution paths in their code. Well, the XSAs quoted above are an indisputable
prove that this is not quite always the case. Realizing that, each developer by
themselves, might be a great step towards a more secure hypervisor...
2. Another security-related aspect of the Xen project is how it totally ignores
problems related to the build process security. Those who don't believe me
should grep the sources for wget, which is now disguised as "FETCHER" shell
variable... (so grep for "FETCHER" string)
I feel embarrassed that I need to explain, at the end of 2015, why the build
process of any serious software project should not blindly download unsigned
components (sources) from the Internet, especially if it is about to execute
Makefiles from these components a moment later... Come on, guys!
(Of course we have been forced to get around this gapping security whole in
Qubes OS [8] ourselves, sadly with a method that is not well suited for
upstreaming).
3. Another thing is, of course: stop adding features to the core hypervisor. We
really need Xen to finally mature, stabilize, and for its development process to
be slowing down over time (just the bug fixes). We need a long-term-supported
hypervisor, which doesn't change with subsonic speed. This would allow this core
code to be widely audited by many experts. If some users want features, these
should perhaps be maintained as additional modules (no, I don't mean dynamically
loaded modules, just compile-time included), preferably in separate repos.
Perhaps also to move all the non-hypervisor code, such as all the toolstacks,
stubdom, etc, into separate repos also. For hygiene, if for nothing else.
Admittedly, some of the features are a result of hardware evolution, such as
e.g. UEFI support. But many are not. Again, maintaining these as optional code
(in separate repos) would be a great step into getting the hypervisor maturing,
finally.
I have already written about it years ago [9], as a matter of fact.
4. Finally, I've been really surprised by the line of reasoning Ian expressed in
the above-mentioned blog post. TL;DR: "we're still doing pretty great, compared
to other projects, because: 1) we have smaller number of publicly disclosed
bugs, and 2) we actually publicly disclose these bugs which we are aware of".
The attitude presented in the blog post is so wrong, that I'm not even sure
where to start commenting on this...
With a single bug like the XSA-148 which, let me repeat that one more time: had
been present in the hypervisor for the last 7 years, so with a bug like this it
really doesn't matter how many (i.e. what number) of critical bugs does the
competition have. Because only one bug of this calibre is enough for the
attacker to never really bother to find another one. The mere fact that
competing hypervisors might got 12 bugs during the same period, really doesn't
make Xen look any better, sorry.
Also, there is really nothing to be proud that you disclose the bugs. It would
be a problem if you didn't.
Hope the above comments might help improve the Xen security. Perhaps some would
perceive them as arrogant or rude. Too bad. Remember the actual attackers will
not be arrogant or rude -- they will just come and exploit bugs, silently.
Admittedly this might not hurt some of the developers ego, not in the short time
at least.
Can we, the Qubes OS project, or myself personally, help with implementing the
above suggestions? Sadly, no. While some of us do contribute occasional patches
to Xen (specifically Marek Marczykowski-Górecki), we really work for a different
project and have different tasks and responsibilities.
Regards,
joanna.
[1] http://xenbits.xen.org/xsa/advisory-148.html
[2] https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-022-2015.txt
[3] https://blog.xenproject.org/2015/10/30/security-vs-features/
[4] https://twitter.com/xen_org/status/660151720463482880
[5] http://xenbits.xen.org/xsa/advisory-109.html
[6] http://xenbits.xen.org/xsa/advisory-123.html
[7] https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-018-2015.txt
[8] https://github.com/QubesOS/qubes-vmm-xen/commit/dcd6c0a4f2c6226a9b706e62469d420579c86975
[9] http://lists.xen.org/archives/html/xen-devel/2013-09/msg01815.html
-----BEGIN PGP SIGNATURE-----
iQIcBAEBAgAGBQJWPOHUAAoJEDOT2L8N3GcYzpEP/An/PTnKDOaC4tPKw3+Y8VIL
n/xIfRPRnPRy18Tbx6EnrKzgohtVvtigtmd/FIxjYVuZ3Luw2B4RFSqUENg758Aa
ANLs4kUD+yaUO82Jfg1nq/6PXBNZlKFovQuYV20LEW9JV6DvMCbzYJ2evZ6t0XS/
EAhUOP1OqY4vb0kah4dwhQKepqwPcD5Tm5LLZn/qbO30e2zN9MkKB851vguQtVIz
k5I8pv+MSQp1efRG2eg470onGtU36IIYFsY1OLihJA9MYh+74FpIA1xoURenJg6+
NJhXEDnxxlz78BJaGOiSwHwB59yd2DXDJKAaUNV/H1LqQu3o1ED+8IZWUARkc0Wl
ckTfQz/++exDhyRcWVHF5GnxEHWdyu/gNZOCNjl4o4HiYS4SQrhTRn7rWwalbyBB
/jG3bAnU8m/Gtp95FtuWXCwuKeeOeBSfnxKMrksxu3JFSNevsYPZu5lrdtUEGLZA
97SwLj70GLesvMqEV3k7XmrQyt8LwyBzLCm+cCocaPEmOQAymeuslrs/RehjGSCQ
L34Ipjvg85GoND64N8X56NuvD+/LrteRhp8hS+aEWv2YpNVJB9tmcOTzLzf7faPK
KPyqa2lW7XKwm7n0WCbtPnrVHRbFPbKvkJRDnfPDwEAEiZcj0SjJ8h7fIDSQ4qac
vgYBTJr/2cRrtC4y1An5
=zQmw
-----END PGP SIGNATURE-----
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Critique of the Xen Security Process
2015-11-06 17:22 Critique of the Xen Security Process Joanna Rutkowska
@ 2015-11-06 19:41 ` James Bulpin
2015-11-06 22:42 ` Low Eel
` (5 subsequent siblings)
6 siblings, 0 replies; 25+ messages in thread
From: James Bulpin @ 2015-11-06 19:41 UTC (permalink / raw
To: Joanna Rutkowska, xen-devel@lists.xen.org; +Cc: qubes-devel@googlegroups.com
On Fri, Nov 06, 2015, Joanna Rutkowska wrote
> [snip] I was then asked to share some more
> thoughts about how I thought Xen could actually improve its security
> process [4].
Thanks Joanna for taking the time to put these thoughts into writing.
I think there are a number of actionable things here we should be taking
a look at as a community. Some of the key things from my perspective:
1. Fixing the use of the FETCHER in the build process (I know there has
already been some discussion around this as part of the Raisin work). In
the commercial Xen distribution I'm responsible for we actually replace
this mechanism and run in an isolated build environment; I'd like to see
this be the standard model.
2. "Disaggregating" the hypervisor makes a lot of sense. With Xen seeing
growth in embedded and client use-cases as well as the traditional server
and cloud ones it will increasingly be the case that some hypervisor
features are necessary in some cases and not in others. I certainly only
want my TCB to include stuff I really need.
3. Do we need to revisit entire old subsystems/features that have been
subject to years of incremental development? Perhaps a line item for Xen
4.7 could be a holistic review of the PV MM code. HVMLite may help in this
specific area longer term but PV (and some of our other older features)
are going to be around for a while yet.
4. Development/coding guidance and standards. This is something we need
anyway as we grow the community. We should consider more specific
guidance on defensive coding practices.
To maintainers and committers: perhaps some of these architectural
direction topics would be good discussions for an in-person dev meeting.
Cheers,
James
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Critique of the Xen Security Process
2015-11-06 17:22 Critique of the Xen Security Process Joanna Rutkowska
2015-11-06 19:41 ` James Bulpin
@ 2015-11-06 22:42 ` Low Eel
2015-11-07 16:51 ` w.peter.howell
` (4 subsequent siblings)
6 siblings, 0 replies; 25+ messages in thread
From: Low Eel @ 2015-11-06 22:42 UTC (permalink / raw
To: qubes-devel; +Cc: joanna, xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 263 bytes --]
Well exposed, thank you Johanna.
Just one question: anybody knows which EAL level has Xen being certified
for?
The question may seem stupid, but I wonder if the approach of closing
execution path can survive to semiformal or formal verification.
Thanks
L.
[-- Attachment #1.2: Type: text/html, Size: 312 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Critique of the Xen Security Process
2015-11-06 17:22 Critique of the Xen Security Process Joanna Rutkowska
2015-11-06 19:41 ` James Bulpin
2015-11-06 22:42 ` Low Eel
@ 2015-11-07 16:51 ` w.peter.howell
2015-11-09 12:11 ` Jan Beulich
` (3 subsequent siblings)
6 siblings, 0 replies; 25+ messages in thread
From: w.peter.howell @ 2015-11-07 16:51 UTC (permalink / raw
To: qubes-devel; +Cc: joanna, xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 20098 bytes --]
On Friday, November 6, 2015 at 10:24:00 AM UTC-7, joanna wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hello,
>
> Recently Xen has released the XSA-148 advisory [1] addressing a fatal bug
> in the
> hypervisor. The bug has been lurking there for the last 7 years! We, the
> Qubes
> OS Project, have commented on this in our Security Bulletin #22 [2]. And
> far
> from enthusiastic commentary that was (FWIW, it was me who wrote this QSB,
> as
> evidenced in the commits log, in case some from the Xen community would
> like to
> direct their rage towards a particular human being ;) Ian Jackson then
> wrote a
> response on the Xen blog [3]. I was then asked to share some more thoughts
> about
> how I thought Xen could actually improve its security process [4]. So, I
> share
> some these below:
>
> 1. First of all, I wish Xen was somehow more defensively coded. To provide
> some
> examples:
>
> a. In XSA-109 [5] there was a problem with the hypervisor dereferencing a
> NULL
> pointer. The problem was fixed by the Xen Security Team by applying a
> patch
> which (hopefully) made sure the execution path that lead to this NULL
> pointer
> dereferencing code was never taken. Back then I suggested (on the Xen
> pre-disclosure list) to make this patch more explicit though:
>
> > On Wed, Jan 21, 2015 at 02:31:51PM +0100, Joanna Rutkowska wrote:
>
> > (...)
> >>
>
> >> Wouldn't it be prudent to also check if:
>
> >>
>
> >> (v->arch.paging.mode>{write_guest_entry,cmpxchg_guest_entry} != NULL)
>
> >>
>
> >> ... in the two affected functions, just before derefing these function
>
> >> pointers?
>
> >>
>
> >> Going even a step further: how about replacing all
>
> >> function-pointer-based calls with macros that first validates the
>
> >> pointer before derefing it? At least when the system doesn't have SMEP?
>
> >>
>
>
> ...to which I got a reply from one of the Xen Security Team engineers that
> the
> above might perhaps be justified in debug builds only, followed by a
> standard:
> "feel free to contribute a patch".
>
This is scary, but unfortunately, it's commonplace. There is a sad
mentality of not including basic checks because it will generate extra
instructions that will take more microseconds to execute. That may have
been a concern when we were running 1MHz 8-bit processors, but now that
code takes such an insignificant amount of time to execute that it doesn't
matter. Coding should no longer be about making it as fast and efficient
as possible -- it needs to be about making it stable and secure.
> b. The XSA-123 [6] was another critical security bug in Xen, this time
> resulting
> from one of the hypervisor developer's fetish to use an absolutely
> confusing
> construct in order to save a few modest bytes in a structure which might
> have
> been allocated by the system maybe a few tens of times at best. Even more
> worrying was the way how Xen Security Team decided to fix the bug: again
> by
> modifying some condition in the code further up the execution path, with
> the
> hopes that this time they would ensure this puzzling construct would
> always be
> used properly. We wrote more about this in our QSB #18 [7].
>
> c. Finally, the way how Xen fixed the recent XSA-148 looks also very
> reactive,
> IMHO. With a bug of this calibre, I would expect Xen to carefully review
> and
> augment all its PV memory virtualization code with additional checks
> (ASSERTs),
> ensuring certain invariants are always satisfied. Such as e.g. that none
> of the
> pages containing PDEs or PTEs are becoming writeable by the VM.
>
> I can't help but have a feeling that some of the Xen developers seem to be
> overconfident in their belief they can fully understand all the possible
> execution paths in their code. Well, the XSAs quoted above are an
> indisputable
> prove that this is not quite always the case. Realizing that, each
> developer by
> themselves, might be a great step towards a more secure hypervisor...
>
> 2. Another security-related aspect of the Xen project is how it totally
> ignores
> problems related to the build process security. Those who don't believe me
> should grep the sources for wget, which is now disguised as "FETCHER"
> shell
> variable... (so grep for "FETCHER" string)
>
> I feel embarrassed that I need to explain, at the end of 2015, why the
> build
> process of any serious software project should not blindly download
> unsigned
> components (sources) from the Internet, especially if it is about to
> execute
> Makefiles from these components a moment later... Come on, guys!
>
> (Of course we have been forced to get around this gapping security whole
> in
> Qubes OS [8] ourselves, sadly with a method that is not well suited for
> upstreaming).
>
Maybe it's time to split off the code into a new branch specifically for
Qubes, and pick and choose what gets pulled over from their code?
>
> 3. Another thing is, of course: stop adding features to the core
> hypervisor. We
> really need Xen to finally mature, stabilize, and for its development
> process to
> be slowing down over time (just the bug fixes). We need a
> long-term-supported
> hypervisor, which doesn't change with subsonic speed. This would allow
> this core
> code to be widely audited by many experts. If some users want features,
> these
> should perhaps be maintained as additional modules (no, I don't mean
> dynamically
> loaded modules, just compile-time included), preferably in separate repos.
>
> Perhaps also to move all the non-hypervisor code, such as all the
> toolstacks,
> stubdom, etc, into separate repos also. For hygiene, if for nothing else.
>
> Admittedly, some of the features are a result of hardware evolution, such
> as
> e.g. UEFI support. But many are not. Again, maintaining these as optional
> code
> (in separate repos) would be a great step into getting the hypervisor
> maturing,
> finally.
>
> I have already written about it years ago [9], as a matter of fact.
>
> 4. Finally, I've been really surprised by the line of reasoning Ian
> expressed in
> the above-mentioned blog post. TL;DR: "we're still doing pretty great,
> compared
> to other projects, because: 1) we have smaller number of publicly
> disclosed
> bugs, and 2) we actually publicly disclose these bugs which we are aware
> of".
>
> The attitude presented in the blog post is so wrong, that I'm not even
> sure
> where to start commenting on this...
>
The bigger question is how much testing is really being done to find
vulnerabilities in the first place. Xen certainly isn't Windows with a
base of hundreds of millions and thousands trying to rip it apart and break
it. It sounds like we'll be in for more surprises to come.
>
> With a single bug like the XSA-148 which, let me repeat that one more
> time: had
> been present in the hypervisor for the last 7 years, so with a bug like
> this it
> really doesn't matter how many (i.e. what number) of critical bugs does
> the
> competition have. Because only one bug of this calibre is enough for the
> attacker to never really bother to find another one. The mere fact that
> competing hypervisors might got 12 bugs during the same period, really
> doesn't
> make Xen look any better, sorry.
>
> Also, there is really nothing to be proud that you disclose the bugs. It
> would
> be a problem if you didn't.
>
> Hope the above comments might help improve the Xen security. Perhaps some
> would
> perceive them as arrogant or rude. Too bad. Remember the actual attackers
> will
> not be arrogant or rude -- they will just come and exploit bugs, silently.
> Admittedly this might not hurt some of the developers ego, not in the
> short time
> at least.
>
> Can we, the Qubes OS project, or myself personally, help with implementing
> the
> above suggestions? Sadly, no. While some of us do contribute occasional
> patches
> to Xen (specifically Marek Marczykowski-Górecki), we really work for a
> different
> project and have different tasks and responsibilities.
>
> Regards,
> joanna.
>
> [1] http://xenbits.xen.org/xsa/advisory-148.html
> [2]
> https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-022-2015.txt
> [3] https://blog.xenproject.org/2015/10/30/security-vs-features/
> [4] https://twitter.com/xen_org/status/660151720463482880
> [5] http://xenbits.xen.org/xsa/advisory-109.html
> [6] http://xenbits.xen.org/xsa/advisory-123.html
> [7]
> https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-018-2015.txt
> [8]
> https://github.com/QubesOS/qubes-vmm-xen/commit/dcd6c0a4f2c6226a9b706e62469d420579c86975
> [9] http://lists.xen.org/archives/html/xen-devel/2013-09/msg01815.html
> -----BEGIN PGP SIGNATURE-----
>
> iQIcBAEBAgAGBQJWPOHUAAoJEDOT2L8N3GcYzpEP/An/PTnKDOaC4tPKw3+Y8VIL
> n/xIfRPRnPRy18Tbx6EnrKzgohtVvtigtmd/FIxjYVuZ3Luw2B4RFSqUENg758Aa
> ANLs4kUD+yaUO82Jfg1nq/6PXBNZlKFovQuYV20LEW9JV6DvMCbzYJ2evZ6t0XS/
> EAhUOP1OqY4vb0kah4dwhQKepqwPcD5Tm5LLZn/qbO30e2zN9MkKB851vguQtVIz
> k5I8pv+MSQp1efRG2eg470onGtU36IIYFsY1OLihJA9MYh+74FpIA1xoURenJg6+
> NJhXEDnxxlz78BJaGOiSwHwB59yd2DXDJKAaUNV/H1LqQu3o1ED+8IZWUARkc0Wl
> ckTfQz/++exDhyRcWVHF5GnxEHWdyu/gNZOCNjl4o4HiYS4SQrhTRn7rWwalbyBB
> /jG3bAnU8m/Gtp95FtuWXCwuKeeOeBSfnxKMrksxu3JFSNevsYPZu5lrdtUEGLZA
> 97SwLj70GLesvMqEV3k7XmrQyt8LwyBzLCm+cCocaPEmOQAymeuslrs/RehjGSCQ
> L34Ipjvg85GoND64N8X56NuvD+/LrteRhp8hS+aEWv2YpNVJB9tmcOTzLzf7faPK
> KPyqa2lW7XKwm7n0WCbtPnrVHRbFPbKvkJRDnfPDwEAEiZcj0SjJ8h7fIDSQ4qac
> vgYBTJr/2cRrtC4y1An5
> =zQmw
> -----END PGP SIGNATURE-----
>
On Friday, November 6, 2015 at 10:24:00 AM UTC-7, joanna wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hello,
>
> Recently Xen has released the XSA-148 advisory [1] addressing a fatal bug
> in the
> hypervisor. The bug has been lurking there for the last 7 years! We, the
> Qubes
> OS Project, have commented on this in our Security Bulletin #22 [2]. And
> far
> from enthusiastic commentary that was (FWIW, it was me who wrote this QSB,
> as
> evidenced in the commits log, in case some from the Xen community would
> like to
> direct their rage towards a particular human being ;) Ian Jackson then
> wrote a
> response on the Xen blog [3]. I was then asked to share some more thoughts
> about
> how I thought Xen could actually improve its security process [4]. So, I
> share
> some these below:
>
> 1. First of all, I wish Xen was somehow more defensively coded. To provide
> some
> examples:
>
> a. In XSA-109 [5] there was a problem with the hypervisor dereferencing a
> NULL
> pointer. The problem was fixed by the Xen Security Team by applying a
> patch
> which (hopefully) made sure the execution path that lead to this NULL
> pointer
> dereferencing code was never taken. Back then I suggested (on the Xen
> pre-disclosure list) to make this patch more explicit though:
>
> > On Wed, Jan 21, 2015 at 02:31:51PM +0100, Joanna Rutkowska wrote:
>
> > (...)
> >>
>
> >> Wouldn't it be prudent to also check if:
>
> >>
>
> >> (v->arch.paging.mode>{write_guest_entry,cmpxchg_guest_entry} != NULL)
>
> >>
>
> >> ... in the two affected functions, just before derefing these function
>
> >> pointers?
>
> >>
>
> >> Going even a step further: how about replacing all
>
> >> function-pointer-based calls with macros that first validates the
>
> >> pointer before derefing it? At least when the system doesn't have SMEP?
>
> >>
>
>
> ...to which I got a reply from one of the Xen Security Team engineers that
> the
> above might perhaps be justified in debug builds only, followed by a
> standard:
> "feel free to contribute a patch".
>
> b. The XSA-123 [6] was another critical security bug in Xen, this time
> resulting
> from one of the hypervisor developer's fetish to use an absolutely
> confusing
> construct in order to save a few modest bytes in a structure which might
> have
> been allocated by the system maybe a few tens of times at best. Even more
> worrying was the way how Xen Security Team decided to fix the bug: again
> by
> modifying some condition in the code further up the execution path, with
> the
> hopes that this time they would ensure this puzzling construct would
> always be
> used properly. We wrote more about this in our QSB #18 [7].
>
> c. Finally, the way how Xen fixed the recent XSA-148 looks also very
> reactive,
> IMHO. With a bug of this calibre, I would expect Xen to carefully review
> and
> augment all its PV memory virtualization code with additional checks
> (ASSERTs),
> ensuring certain invariants are always satisfied. Such as e.g. that none
> of the
> pages containing PDEs or PTEs are becoming writeable by the VM.
>
> I can't help but have a feeling that some of the Xen developers seem to be
> overconfident in their belief they can fully understand all the possible
> execution paths in their code. Well, the XSAs quoted above are an
> indisputable
> prove that this is not quite always the case. Realizing that, each
> developer by
> themselves, might be a great step towards a more secure hypervisor...
>
> 2. Another security-related aspect of the Xen project is how it totally
> ignores
> problems related to the build process security. Those who don't believe me
> should grep the sources for wget, which is now disguised as "FETCHER"
> shell
> variable... (so grep for "FETCHER" string)
>
> I feel embarrassed that I need to explain, at the end of 2015, why the
> build
> process of any serious software project should not blindly download
> unsigned
> components (sources) from the Internet, especially if it is about to
> execute
> Makefiles from these components a moment later... Come on, guys!
>
> (Of course we have been forced to get around this gapping security whole
> in
> Qubes OS [8] ourselves, sadly with a method that is not well suited for
> upstreaming).
>
> 3. Another thing is, of course: stop adding features to the core
> hypervisor. We
> really need Xen to finally mature, stabilize, and for its development
> process to
> be slowing down over time (just the bug fixes). We need a
> long-term-supported
> hypervisor, which doesn't change with subsonic speed. This would allow
> this core
> code to be widely audited by many experts. If some users want features,
> these
> should perhaps be maintained as additional modules (no, I don't mean
> dynamically
> loaded modules, just compile-time included), preferably in separate repos.
>
> Perhaps also to move all the non-hypervisor code, such as all the
> toolstacks,
> stubdom, etc, into separate repos also. For hygiene, if for nothing else.
>
> Admittedly, some of the features are a result of hardware evolution, such
> as
> e.g. UEFI support. But many are not. Again, maintaining these as optional
> code
> (in separate repos) would be a great step into getting the hypervisor
> maturing,
> finally.
>
> I have already written about it years ago [9], as a matter of fact.
>
> 4. Finally, I've been really surprised by the line of reasoning Ian
> expressed in
> the above-mentioned blog post. TL;DR: "we're still doing pretty great,
> compared
> to other projects, because: 1) we have smaller number of publicly
> disclosed
> bugs, and 2) we actually publicly disclose these bugs which we are aware
> of".
>
> The attitude presented in the blog post is so wrong, that I'm not even
> sure
> where to start commenting on this...
>
> With a single bug like the XSA-148 which, let me repeat that one more
> time: had
> been present in the hypervisor for the last 7 years, so with a bug like
> this it
> really doesn't matter how many (i.e. what number) of critical bugs does
> the
> competition have. Because only one bug of this calibre is enough for the
> attacker to never really bother to find another one. The mere fact that
> competing hypervisors might got 12 bugs during the same period, really
> doesn't
> make Xen look any better, sorry.
>
> Also, there is really nothing to be proud that you disclose the bugs. It
> would
> be a problem if you didn't.
>
> Hope the above comments might help improve the Xen security. Perhaps some
> would
> perceive them as arrogant or rude. Too bad. Remember the actual attackers
> will
> not be arrogant or rude -- they will just come and exploit bugs, silently.
> Admittedly this might not hurt some of the developers ego, not in the
> short time
> at least.
>
> Can we, the Qubes OS project, or myself personally, help with implementing
> the
> above suggestions? Sadly, no. While some of us do contribute occasional
> patches
> to Xen (specifically Marek Marczykowski-Górecki), we really work for a
> different
> project and have different tasks and responsibilities.
>
> Regards,
> joanna.
>
> [1] http://xenbits.xen.org/xsa/advisory-148.html
> [2]
> https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-022-2015.txt
> [3] https://blog.xenproject.org/2015/10/30/security-vs-features/
> [4] https://twitter.com/xen_org/status/660151720463482880
> [5] http://xenbits.xen.org/xsa/advisory-109.html
> [6] http://xenbits.xen.org/xsa/advisory-123.html
> [7]
> https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-018-2015.txt
> [8]
> https://github.com/QubesOS/qubes-vmm-xen/commit/dcd6c0a4f2c6226a9b706e62469d420579c86975
> [9] http://lists.xen.org/archives/html/xen-devel/2013-09/msg01815.html
> -----BEGIN PGP SIGNATURE-----
>
> iQIcBAEBAgAGBQJWPOHUAAoJEDOT2L8N3GcYzpEP/An/PTnKDOaC4tPKw3+Y8VIL
> n/xIfRPRnPRy18Tbx6EnrKzgohtVvtigtmd/FIxjYVuZ3Luw2B4RFSqUENg758Aa
> ANLs4kUD+yaUO82Jfg1nq/6PXBNZlKFovQuYV20LEW9JV6DvMCbzYJ2evZ6t0XS/
> EAhUOP1OqY4vb0kah4dwhQKepqwPcD5Tm5LLZn/qbO30e2zN9MkKB851vguQtVIz
> k5I8pv+MSQp1efRG2eg470onGtU36IIYFsY1OLihJA9MYh+74FpIA1xoURenJg6+
> NJhXEDnxxlz78BJaGOiSwHwB59yd2DXDJKAaUNV/H1LqQu3o1ED+8IZWUARkc0Wl
> ckTfQz/++exDhyRcWVHF5GnxEHWdyu/gNZOCNjl4o4HiYS4SQrhTRn7rWwalbyBB
> /jG3bAnU8m/Gtp95FtuWXCwuKeeOeBSfnxKMrksxu3JFSNevsYPZu5lrdtUEGLZA
> 97SwLj70GLesvMqEV3k7XmrQyt8LwyBzLCm+cCocaPEmOQAymeuslrs/RehjGSCQ
> L34Ipjvg85GoND64N8X56NuvD+/LrteRhp8hS+aEWv2YpNVJB9tmcOTzLzf7faPK
> KPyqa2lW7XKwm7n0WCbtPnrVHRbFPbKvkJRDnfPDwEAEiZcj0SjJ8h7fIDSQ4qac
> vgYBTJr/2cRrtC4y1An5
> =zQmw
> -----END PGP SIGNATURE-----
>
[-- Attachment #1.2: Type: text/html, Size: 31635 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Critique of the Xen Security Process
2015-11-06 17:22 Critique of the Xen Security Process Joanna Rutkowska
` (2 preceding siblings ...)
2015-11-07 16:51 ` w.peter.howell
@ 2015-11-09 12:11 ` Jan Beulich
2015-11-09 16:31 ` [qubes-devel] " Franz
2015-11-09 21:48 ` Doug Goldstein
` (2 subsequent siblings)
6 siblings, 1 reply; 25+ messages in thread
From: Jan Beulich @ 2015-11-09 12:11 UTC (permalink / raw
To: Joanna Rutkowska; +Cc: xen-devel, qubes-devel
>>> On 06.11.15 at 18:22, <joanna@invisiblethingslab.com> wrote:
> 1. First of all, I wish Xen was somehow more defensively coded. To provide
> some
> examples:
>
> a. In XSA-109 [5] there was a problem with the hypervisor dereferencing a
> NULL
> pointer. The problem was fixed by the Xen Security Team by applying a patch
> which (hopefully) made sure the execution path that lead to this NULL
> pointer
> dereferencing code was never taken. Back then I suggested (on the Xen
> pre-disclosure list) to make this patch more explicit though:
>
>> On Wed, Jan 21, 2015 at 02:31:51PM +0100, Joanna Rutkowska wrote:
>
>> (...)
>>>
>>> Wouldn't it be prudent to also check if:
>>>
>>> (v->arch.paging.mode>{write_guest_entry,cmpxchg_guest_entry} != NULL)
>>>
>>> ... in the two affected functions, just before derefing these function
>>> pointers?
>>>
>>> Going even a step further: how about replacing all
>>> function-pointer-based calls with macros that first validates the
>>> pointer before derefing it? At least when the system doesn't have SMEP?
>>>
> ...to which I got a reply from one of the Xen Security Team engineers that
> the
> above might perhaps be justified in debug builds only, followed by a
> standard:
> "feel free to contribute a patch".
And you know what - I can only repeat this now. Indeed, the points
you make are all quite reasonable from a technical perspective (and
especially the code quality aspect is something that I also have been
complaining _but also trying to do something_ about repeatedly), but
the attitude behind the way you present them bothers me quite a lot:
I get the impression that you (a) expect others to do work that you
neither pay for, nor are at the slightest willing to help with and (b)
de-value all our efforts during the last so many years to move the
code base from a research project to a production usable one.
Yes, the work item you mention above has been on our virtual,
combined todo list, but with all the other work that needs to be done
it seems like no-one actually found the time to actually carry this out.
A good example that just pointing out issues is nice and easy, but
actually finding time to do the work is quite a bit harder.
> b. The XSA-123 [6] was another critical security bug in Xen, this time
> resulting
> from one of the hypervisor developer's fetish to use an absolutely confusing
> construct in order to save a few modest bytes in a structure which might
> have
> been allocated by the system maybe a few tens of times at best. Even more
> worrying was the way how Xen Security Team decided to fix the bug: again by
> modifying some condition in the code further up the execution path, with the
> hopes that this time they would ensure this puzzling construct would always
> be
> used properly. We wrote more about this in our QSB #18 [7].
I suppose you already know what my response here would be?
> c. Finally, the way how Xen fixed the recent XSA-148 looks also very reactive,
> IMHO. With a bug of this calibre, I would expect Xen to carefully review and
> augment all its PV memory virtualization code with additional checks (ASSERTs),
> ensuring certain invariants are always satisfied. Such as e.g. that none of the
> pages containing PDEs or PTEs are becoming writeable by the VM.
Perhaps I'm overlooking something, but I cannot see what kinds of
ASSERT()s could have helped avoid that problem. The only things I
can see that would have helped here are
- better thinking through of all cases by the original developer
- better review before things going in
- regular code audits
None of which, according to what you write further down, you're
willing to help with.
> I can't help but have a feeling that some of the Xen developers seem to be
> overconfident in their belief they can fully understand all the possible
> execution paths in their code. Well, the XSAs quoted above are an
> indisputable
> prove that this is not quite always the case. Realizing that, each developer
> by
> themselves, might be a great step towards a more secure hypervisor...
I think we need to find a reasonable balance here. Taking your
statement to the extreme, we might end up with code having more
ASSERT()s in it than anything else. Surely that can't be the goal,
as this would severely hamper maintainability. It simply doesn't
make sense for every function to mistrust its callers in all aspects.
> 2. Another security-related aspect of the Xen project is how it totally
> ignores
> problems related to the build process security. Those who don't believe me
> should grep the sources for wget, which is now disguised as "FETCHER" shell
> variable... (so grep for "FETCHER" string)
>
> I feel embarrassed that I need to explain, at the end of 2015, why the build
> process of any serious software project should not blindly download unsigned
> components (sources) from the Internet, especially if it is about to execute
> Makefiles from these components a moment later... Come on, guys!
>
> (Of course we have been forced to get around this gapping security whole in
> Qubes OS [8] ourselves, sadly with a method that is not well suited for
> upstreaming).
To be honest, for a purely development oriented build environment
this might by fine. No-one is required to use that, or even run their
builds outside of sandboxes. Even in my local build environment
(used for just my own purposes) I avoid any such downloads, albeit
perhaps more for performance than security reasons. But what this
tells us is that avoiding to build on this "convenience" is quite simple.
> 3. Another thing is, of course: stop adding features to the core hypervisor. We
> really need Xen to finally mature, stabilize, and for its development process to
> be slowing down over time (just the bug fixes). We need a long-term-supported
> hypervisor, which doesn't change with subsonic speed. This would allow this core
> code to be widely audited by many experts. If some users want features, these
> should perhaps be maintained as additional modules (no, I don't mean dynamically
> loaded modules, just compile-time included), preferably in separate repos.
I admit that, especially during the last couple of months, I have
wished the ratio between feature addition patches and bug fixes
would shift towards the latter (and that without hoping for there
to be more bugs to be fixed - in fact the opposite has seemed to
be the case in the more recent past), albeit for other reasons than
yours.
But is not adding new features realistic? I don't think so. People
want a secure _and_ functionally complete _and_ performant
hypervisor, and the latter two require making use of new
hardware features as they become available (and occasionally
introduce pure software features too, see the xSplice work
underway).
> Admittedly, some of the features are a result of hardware evolution, such as
> e.g. UEFI support. But many are not. Again, maintaining these as optional code
> (in separate repos) would be a great step into getting the hypervisor maturing,
> finally.
This, afaict, would quickly become unwieldy and unmaintainable.
> 4. Finally, I've been really surprised by the line of reasoning Ian
> expressed in
> the above-mentioned blog post. TL;DR: "we're still doing pretty great,
> compared
> to other projects, because: 1) we have smaller number of publicly disclosed
> bugs, and 2) we actually publicly disclose these bugs which we are aware
> of".
>
> The attitude presented in the blog post is so wrong, that I'm not even sure
> where to start commenting on this...
>
> With a single bug like the XSA-148 which, let me repeat that one more time: had
> been present in the hypervisor for the last 7 years, so with a bug like this it
> really doesn't matter how many (i.e. what number) of critical bugs does the
> competition have.
And let me repeat - it doesn't matter for how long the bug has been
there without anyone noticing. Had you noticed it 7 years ago, and
had we both rejected your fixes and not taken steps to fix it
ourselves, I could see you being upset. But you having done nothing
(Out of the now around 150 security issues in Xen, how many did you
discover, btw?), what gives you the right to complain about a bug (no
matter how severe) having been there for an admittedly rather long
time? If you weren't able to spot it, how can you demand someone
else to be able to? It is the nature of bugs that at times they hide
rather well.
> Can we, the Qubes OS project, or myself personally, help with implementing the
> above suggestions? Sadly, no. While some of us do contribute occasional patches
> to Xen (specifically Marek Marczykowski-Górecki), we really work for a different
> project and have different tasks and responsibilities.
But you seem to forget that many if not most of us too are not
employed by their employers to look after Xen as their sole or
primary task. I.e. if everyone reasoned this way, we could simply
bin the entire project.
So bottom line - _constructive_ criticism is welcome, but
accusations, especially without any intent to help changing things
(other than demanding others to do the necessary work), aren't.
(Please note that all of the above are my personal views, i.e. I'm
explicitly not speaking on behalf of the project.)
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [qubes-devel] Re: Critique of the Xen Security Process
2015-11-09 12:11 ` Jan Beulich
@ 2015-11-09 16:31 ` Franz
2015-11-09 18:15 ` Wojtek Porczyk
2015-11-10 10:52 ` Lars Kurth
0 siblings, 2 replies; 25+ messages in thread
From: Franz @ 2015-11-09 16:31 UTC (permalink / raw
To: Jan Beulich; +Cc: xen-devel, Joanna Rutkowska, qubes-devel@googlegroups.com
[-- Attachment #1.1: Type: text/plain, Size: 11355 bytes --]
On Mon, Nov 9, 2015 at 12:11 PM, Jan Beulich <JBeulich@suse.com> wrote:
> >>> On 06.11.15 at 18:22, <joanna@invisiblethingslab.com> wrote:
> > 1. First of all, I wish Xen was somehow more defensively coded. To
> provide
> > some
> > examples:
> >
> > a. In XSA-109 [5] there was a problem with the hypervisor dereferencing a
> > NULL
> > pointer. The problem was fixed by the Xen Security Team by applying a
> patch
> > which (hopefully) made sure the execution path that lead to this NULL
> > pointer
> > dereferencing code was never taken. Back then I suggested (on the Xen
> > pre-disclosure list) to make this patch more explicit though:
> >
> >> On Wed, Jan 21, 2015 at 02:31:51PM +0100, Joanna Rutkowska wrote:
> >
> >> (...)
> >>>
> >>> Wouldn't it be prudent to also check if:
> >>>
> >>> (v->arch.paging.mode>{write_guest_entry,cmpxchg_guest_entry} != NULL)
> >>>
> >>> ... in the two affected functions, just before derefing these function
> >>> pointers?
> >>>
> >>> Going even a step further: how about replacing all
> >>> function-pointer-based calls with macros that first validates the
> >>> pointer before derefing it? At least when the system doesn't have SMEP?
> >>>
> > ...to which I got a reply from one of the Xen Security Team engineers
> that
> > the
> > above might perhaps be justified in debug builds only, followed by a
> > standard:
> > "feel free to contribute a patch".
>
> And you know what - I can only repeat this now. Indeed, the points
> you make are all quite reasonable from a technical perspective (and
> especially the code quality aspect is something that I also have been
> complaining _but also trying to do something_ about repeatedly), but
> the attitude behind the way you present them bothers me quite a lot:
> I get the impression that you (a) expect others to do work that you
> neither pay for, nor are at the slightest willing to help with and (b)
> de-value all our efforts during the last so many years to move the
> code base from a research project to a production usable one.
>
> Yes, the work item you mention above has been on our virtual,
> combined todo list, but with all the other work that needs to be done
> it seems like no-one actually found the time to actually carry this out.
> A good example that just pointing out issues is nice and easy, but
> actually finding time to do the work is quite a bit harder.
>
> > b. The XSA-123 [6] was another critical security bug in Xen, this time
> > resulting
> > from one of the hypervisor developer's fetish to use an absolutely
> confusing
> > construct in order to save a few modest bytes in a structure which might
> > have
> > been allocated by the system maybe a few tens of times at best. Even more
> > worrying was the way how Xen Security Team decided to fix the bug: again
> by
> > modifying some condition in the code further up the execution path, with
> the
> > hopes that this time they would ensure this puzzling construct would
> always
> > be
> > used properly. We wrote more about this in our QSB #18 [7].
>
> I suppose you already know what my response here would be?
>
> > c. Finally, the way how Xen fixed the recent XSA-148 looks also very
> reactive,
> > IMHO. With a bug of this calibre, I would expect Xen to carefully review
> and
> > augment all its PV memory virtualization code with additional checks
> (ASSERTs),
> > ensuring certain invariants are always satisfied. Such as e.g. that none
> of the
> > pages containing PDEs or PTEs are becoming writeable by the VM.
>
> Perhaps I'm overlooking something, but I cannot see what kinds of
> ASSERT()s could have helped avoid that problem. The only things I
> can see that would have helped here are
> - better thinking through of all cases by the original developer
> - better review before things going in
> - regular code audits
> None of which, according to what you write further down, you're
> willing to help with.
>
> > I can't help but have a feeling that some of the Xen developers seem to
> be
> > overconfident in their belief they can fully understand all the possible
> > execution paths in their code. Well, the XSAs quoted above are an
> > indisputable
> > prove that this is not quite always the case. Realizing that, each
> developer
> > by
> > themselves, might be a great step towards a more secure hypervisor...
>
> I think we need to find a reasonable balance here. Taking your
> statement to the extreme, we might end up with code having more
> ASSERT()s in it than anything else. Surely that can't be the goal,
> as this would severely hamper maintainability. It simply doesn't
> make sense for every function to mistrust its callers in all aspects.
>
> > 2. Another security-related aspect of the Xen project is how it totally
> > ignores
> > problems related to the build process security. Those who don't believe
> me
> > should grep the sources for wget, which is now disguised as "FETCHER"
> shell
> > variable... (so grep for "FETCHER" string)
> >
> > I feel embarrassed that I need to explain, at the end of 2015, why the
> build
> > process of any serious software project should not blindly download
> unsigned
> > components (sources) from the Internet, especially if it is about to
> execute
> > Makefiles from these components a moment later... Come on, guys!
> >
> > (Of course we have been forced to get around this gapping security whole
> in
> > Qubes OS [8] ourselves, sadly with a method that is not well suited for
> > upstreaming).
>
> To be honest, for a purely development oriented build environment
> this might by fine. No-one is required to use that, or even run their
> builds outside of sandboxes. Even in my local build environment
> (used for just my own purposes) I avoid any such downloads, albeit
> perhaps more for performance than security reasons. But what this
> tells us is that avoiding to build on this "convenience" is quite simple.
>
> > 3. Another thing is, of course: stop adding features to the core
> hypervisor. We
> > really need Xen to finally mature, stabilize, and for its development
> process to
> > be slowing down over time (just the bug fixes). We need a
> long-term-supported
> > hypervisor, which doesn't change with subsonic speed. This would allow
> this core
> > code to be widely audited by many experts. If some users want features,
> these
> > should perhaps be maintained as additional modules (no, I don't mean
> dynamically
> > loaded modules, just compile-time included), preferably in separate
> repos.
>
> I admit that, especially during the last couple of months, I have
> wished the ratio between feature addition patches and bug fixes
> would shift towards the latter (and that without hoping for there
> to be more bugs to be fixed - in fact the opposite has seemed to
> be the case in the more recent past), albeit for other reasons than
> yours.
>
> But is not adding new features realistic? I don't think so. People
> want a secure _and_ functionally complete _and_ performant
> hypervisor, and the latter two require making use of new
> hardware features as they become available (and occasionally
> introduce pure software features too, see the xSplice work
> underway).
>
> > Admittedly, some of the features are a result of hardware evolution,
> such as
> > e.g. UEFI support. But many are not. Again, maintaining these as
> optional code
> > (in separate repos) would be a great step into getting the hypervisor
> maturing,
> > finally.
>
> This, afaict, would quickly become unwieldy and unmaintainable.
>
> > 4. Finally, I've been really surprised by the line of reasoning Ian
> > expressed in
> > the above-mentioned blog post. TL;DR: "we're still doing pretty great,
> > compared
> > to other projects, because: 1) we have smaller number of publicly
> disclosed
> > bugs, and 2) we actually publicly disclose these bugs which we are aware
> > of".
> >
> > The attitude presented in the blog post is so wrong, that I'm not even
> sure
> > where to start commenting on this...
> >
> > With a single bug like the XSA-148 which, let me repeat that one more
> time: had
> > been present in the hypervisor for the last 7 years, so with a bug like
> this it
> > really doesn't matter how many (i.e. what number) of critical bugs does
> the
> > competition have.
>
> And let me repeat - it doesn't matter for how long the bug has been
> there without anyone noticing. Had you noticed it 7 years ago, and
> had we both rejected your fixes and not taken steps to fix it
> ourselves, I could see you being upset. But you having done nothing
> (Out of the now around 150 security issues in Xen, how many did you
> discover, btw?), what gives you the right to complain about a bug (no
> matter how severe) having been there for an admittedly rather long
> time? If you weren't able to spot it, how can you demand someone
> else to be able to? It is the nature of bugs that at times they hide
> rather well.
>
> > Can we, the Qubes OS project, or myself personally, help with
> implementing the
> > above suggestions? Sadly, no. While some of us do contribute occasional
> patches
> > to Xen (specifically Marek Marczykowski-Górecki), we really work for a
> different
> > project and have different tasks and responsibilities.
>
> But you seem to forget that many if not most of us too are not
> employed by their employers to look after Xen as their sole or
> primary task. I.e. if everyone reasoned this way, we could simply
> bin the entire project.
>
> So bottom line - _constructive_ criticism is welcome, but
> accusations, especially without any intent to help changing things
> (other than demanding others to do the necessary work), aren't.
>
> (Please note that all of the above are my personal views, i.e. I'm
> explicitly not speaking on behalf of the project.)
>
>
It seems positions are antithetic with no possible compromise in view.
Sadly this is a general problem of FOSS software: developers tend to do
what they like and not what users request. And who can blame developers for
that? After all they are working as volunteers so they deserve to do what
they like. No possible compromise.
Perhaps a way out of this impasse is to put bounties on Xen security tasks
identified by Joanna and properly advertise these bounties to Xen users. I
would give particular attention to the companies that use Xen. I imagine
there are many of them. Some of them may already be giving money to Xen. If
some of you knows who are these companies and how is the proper way to set
up a bounty, I offer to write the copywriting letter, already have some
clear ideas how to do that.
Best
Fran
> Jan
>
> --
> You received this message because you are subscribed to the Google Groups
> "qubes-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to qubes-devel+unsubscribe@googlegroups.com.
> To post to this group, send email to qubes-devel@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/qubes-devel/56409B7702000078000B3018%40prv-mh.provo.novell.com
> .
> For more options, visit https://groups.google.com/d/optout.
>
[-- Attachment #1.2: Type: text/html, Size: 13348 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [qubes-devel] Re: Critique of the Xen Security Process
2015-11-09 16:31 ` [qubes-devel] " Franz
@ 2015-11-09 18:15 ` Wojtek Porczyk
2015-11-10 13:09 ` Lars Kurth
2015-11-10 10:52 ` Lars Kurth
1 sibling, 1 reply; 25+ messages in thread
From: Wojtek Porczyk @ 2015-11-09 18:15 UTC (permalink / raw
To: Franz
Cc: xen-devel, Joanna Rutkowska, Jan Beulich,
qubes-devel@googlegroups.com
[-- Attachment #1.1: Type: text/plain, Size: 1302 bytes --]
On Mon, Nov 09, 2015 at 04:31:58PM +0000, Franz wrote:
> Perhaps a way out of this impasse is to put bounties on Xen security tasks
> identified by Joanna and properly advertise these bounties to Xen users.
> [snip]
This is fundamentaly wrong idea. Security isn't something you can
"apply" or put bounty on. It's a state of the mind, especcialy
developer's. Joanna wrote in her mail:
> > > I can't help but have a feeling that some of the Xen developers seem to be
> > > overconfident in their belief they can fully understand all the possible
> > > execution paths in their code. Well, the XSAs quoted above are an indisputable
> > > prove that this is not quite always the case. Realizing that, each developer by
> > > themselves, might be a great step towards a more secure hypervisor...
And that's why we can't just "submit a patch" to "contribute security".
There is something wrong with Xen as a whole project, but that something
isn't the code. There is a mindset to be fixed.
--
regards, _.-._
Wojtek Porczyk .-^' '^-.
Invisible Things Lab |'-.-^-.-'|
| | | |
I do not fear computers, | '-.-' |
I fear lack of them. '-._ : ,-'
-- Isaac Asimov `^-^-_>
[-- Attachment #1.2: Type: application/pgp-signature, Size: 819 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Critique of the Xen Security Process
2015-11-06 17:22 Critique of the Xen Security Process Joanna Rutkowska
` (3 preceding siblings ...)
2015-11-09 12:11 ` Jan Beulich
@ 2015-11-09 21:48 ` Doug Goldstein
2015-11-11 9:43 ` Ian Campbell
2015-11-11 12:34 ` Wei Liu
2015-11-09 22:00 ` chris
2015-11-11 12:59 ` Stefano Stabellini
6 siblings, 2 replies; 25+ messages in thread
From: Doug Goldstein @ 2015-11-09 21:48 UTC (permalink / raw
To: xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 9290 bytes --]
On 11/6/15 11:22 AM, Joanna Rutkowska wrote:
> Hello,
>
> Recently Xen has released the XSA-148 advisory [1] addressing a fatal bug in the
> hypervisor. The bug has been lurking there for the last 7 years! We, the Qubes
> OS Project, have commented on this in our Security Bulletin #22 [2]. And far
> from enthusiastic commentary that was (FWIW, it was me who wrote this QSB, as
> evidenced in the commits log, in case some from the Xen community would like to
> direct their rage towards a particular human being ;) Ian Jackson then wrote a
> response on the Xen blog [3]. I was then asked to share some more thoughts about
> how I thought Xen could actually improve its security process [4]. So, I share
> some these below:
>
> 1. First of all, I wish Xen was somehow more defensively coded. To provide some
> examples:
>
> a. In XSA-109 [5] there was a problem with the hypervisor dereferencing a NULL
> pointer. The problem was fixed by the Xen Security Team by applying a patch
> which (hopefully) made sure the execution path that lead to this NULL pointer
> dereferencing code was never taken. Back then I suggested (on the Xen
> pre-disclosure list) to make this patch more explicit though:
>
>> On Wed, Jan 21, 2015 at 02:31:51PM +0100, Joanna Rutkowska wrote:
>> (...)
>>>
>>> Wouldn't it be prudent to also check if:
>>>
>>> (v->arch.paging.mode>{write_guest_entry,cmpxchg_guest_entry} != NULL)
>>>
>>> ... in the two affected functions, just before derefing these function
>>> pointers?
>>>
>>> Going even a step further: how about replacing all
>>> function-pointer-based calls with macros that first validates the
>>> pointer before derefing it? At least when the system doesn't have SMEP?
>>>
>
> ...to which I got a reply from one of the Xen Security Team engineers that the
> above might perhaps be justified in debug builds only, followed by a standard:
> "feel free to contribute a patch".
>
> b. The XSA-123 [6] was another critical security bug in Xen, this time resulting
> from one of the hypervisor developer's fetish to use an absolutely confusing
> construct in order to save a few modest bytes in a structure which might have
> been allocated by the system maybe a few tens of times at best. Even more
> worrying was the way how Xen Security Team decided to fix the bug: again by
> modifying some condition in the code further up the execution path, with the
> hopes that this time they would ensure this puzzling construct would always be
> used properly. We wrote more about this in our QSB #18 [7].
>
> c. Finally, the way how Xen fixed the recent XSA-148 looks also very reactive,
> IMHO. With a bug of this calibre, I would expect Xen to carefully review and
> augment all its PV memory virtualization code with additional checks (ASSERTs),
> ensuring certain invariants are always satisfied. Such as e.g. that none of the
> pages containing PDEs or PTEs are becoming writeable by the VM.
>
> I can't help but have a feeling that some of the Xen developers seem to be
> overconfident in their belief they can fully understand all the possible
> execution paths in their code. Well, the XSAs quoted above are an indisputable
> prove that this is not quite always the case. Realizing that, each developer by
> themselves, might be a great step towards a more secure hypervisor...
>
> 2. Another security-related aspect of the Xen project is how it totally ignores
> problems related to the build process security. Those who don't believe me
> should grep the sources for wget, which is now disguised as "FETCHER" shell
> variable... (so grep for "FETCHER" string)
>
> I feel embarrassed that I need to explain, at the end of 2015, why the build
> process of any serious software project should not blindly download unsigned
> components (sources) from the Internet, especially if it is about to execute
> Makefiles from these components a moment later... Come on, guys!
I'll echo this sentiment as well. Most distro packagers will dislike
this and need to work around some of this behavior in their respective
distros. Project Raisin is aiming to help with this but it doesn't seem
to have a lot of community effort behind it and it too attempts to
install dependencies on my machine and wants to be run with sudo.
>
> (Of course we have been forced to get around this gapping security whole in
> Qubes OS [8] ourselves, sadly with a method that is not well suited for
> upstreaming).
>
> 3. Another thing is, of course: stop adding features to the core hypervisor. We
> really need Xen to finally mature, stabilize, and for its development process to
> be slowing down over time (just the bug fixes). We need a long-term-supported
> hypervisor, which doesn't change with subsonic speed. This would allow this core
> code to be widely audited by many experts. If some users want features, these
> should perhaps be maintained as additional modules (no, I don't mean dynamically
> loaded modules, just compile-time included), preferably in separate repos.
>
As far as the compile time support goes I'm aiming for this as well.
I've been working on Kconfig support and hope to get that pushed soon
and the idea being that less mature features can be kept off until
they're ready to be default on. The downstream distro/product makers can
then fiddle with these flags to get the hypervisor that fits their needs.
> Perhaps also to move all the non-hypervisor code, such as all the toolstacks,
> stubdom, etc, into separate repos also. For hygiene, if for nothing else.
From a real high level things should be broken into at least 4 repos:
1) hypervisor
2) libxl
3) userland tools (xl and support scripts)
4) stubdom
>
> Admittedly, some of the features are a result of hardware evolution, such as
> e.g. UEFI support. But many are not. Again, maintaining these as optional code
> (in separate repos) would be a great step into getting the hypervisor maturing,
> finally.
>
> I have already written about it years ago [9], as a matter of fact.
>
> 4. Finally, I've been really surprised by the line of reasoning Ian expressed in
> the above-mentioned blog post. TL;DR: "we're still doing pretty great, compared
> to other projects, because: 1) we have smaller number of publicly disclosed
> bugs, and 2) we actually publicly disclose these bugs which we are aware of".
>
> The attitude presented in the blog post is so wrong, that I'm not even sure
> where to start commenting on this...
>
> With a single bug like the XSA-148 which, let me repeat that one more time: had
> been present in the hypervisor for the last 7 years, so with a bug like this it
> really doesn't matter how many (i.e. what number) of critical bugs does the
> competition have. Because only one bug of this calibre is enough for the
> attacker to never really bother to find another one. The mere fact that
> competing hypervisors might got 12 bugs during the same period, really doesn't
> make Xen look any better, sorry.
>
> Also, there is really nothing to be proud that you disclose the bugs. It would
> be a problem if you didn't.
>
> Hope the above comments might help improve the Xen security. Perhaps some would
> perceive them as arrogant or rude. Too bad. Remember the actual attackers will
> not be arrogant or rude -- they will just come and exploit bugs, silently.
> Admittedly this might not hurt some of the developers ego, not in the short time
> at least.
>
> Can we, the Qubes OS project, or myself personally, help with implementing the
> above suggestions? Sadly, no. While some of us do contribute occasional patches
> to Xen (specifically Marek Marczykowski-Górecki), we really work for a different
> project and have different tasks and responsibilities.
This is where I'll disagree with you. I too work for a different project
and have different tasks and responsibilities but as a consumer of Xen
we do have the responsibility to try to upstream things where
applicable. Could create patches locally to disable features that I
don't want/need on? Of course and it probably would take me less time
but in the long run upstreaming a generic framework will have an immense
pay off for not just me and my organization but everyone else. You
cannot rail at a project for not doing the work which serves your end
goals without doing some of that work yourself.
>
> Regards,
> joanna.
>
> [1] http://xenbits.xen.org/xsa/advisory-148.html
> [2] https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-022-2015.txt
> [3] https://blog.xenproject.org/2015/10/30/security-vs-features/
> [4] https://twitter.com/xen_org/status/660151720463482880
> [5] http://xenbits.xen.org/xsa/advisory-109.html
> [6] http://xenbits.xen.org/xsa/advisory-123.html
> [7] https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-018-2015.txt
> [8] https://github.com/QubesOS/qubes-vmm-xen/commit/dcd6c0a4f2c6226a9b706e62469d420579c86975
> [9] http://lists.xen.org/archives/html/xen-devel/2013-09/msg01815.html
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>
--
Doug Goldstein
[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 959 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Critique of the Xen Security Process
2015-11-06 17:22 Critique of the Xen Security Process Joanna Rutkowska
` (4 preceding siblings ...)
2015-11-09 21:48 ` Doug Goldstein
@ 2015-11-09 22:00 ` chris
2015-11-10 2:46 ` [qubes-devel] " Radoslaw Szkodzinski
2015-11-11 12:59 ` Stefano Stabellini
6 siblings, 1 reply; 25+ messages in thread
From: chris @ 2015-11-09 22:00 UTC (permalink / raw
To: Joanna Rutkowska, xen-devel@lists.xen.org, qubes-devel
[-- Attachment #1.1: Type: text/plain, Size: 9005 bytes --]
+1... so many great points here that ive thought many times its almost as
if i could have written it
great post!
chris
On Fri, Nov 6, 2015 at 12:22 PM, Joanna Rutkowska <
joanna@invisiblethingslab.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hello,
>
> Recently Xen has released the XSA-148 advisory [1] addressing a fatal bug
> in the
> hypervisor. The bug has been lurking there for the last 7 years! We, the
> Qubes
> OS Project, have commented on this in our Security Bulletin #22 [2]. And
> far
> from enthusiastic commentary that was (FWIW, it was me who wrote this QSB,
> as
> evidenced in the commits log, in case some from the Xen community would
> like to
> direct their rage towards a particular human being ;) Ian Jackson then
> wrote a
> response on the Xen blog [3]. I was then asked to share some more thoughts
> about
> how I thought Xen could actually improve its security process [4]. So, I
> share
> some these below:
>
> 1. First of all, I wish Xen was somehow more defensively coded. To provide
> some
> examples:
>
> a. In XSA-109 [5] there was a problem with the hypervisor dereferencing a
> NULL
> pointer. The problem was fixed by the Xen Security Team by applying a patch
> which (hopefully) made sure the execution path that lead to this NULL
> pointer
> dereferencing code was never taken. Back then I suggested (on the Xen
> pre-disclosure list) to make this patch more explicit though:
>
> > On Wed, Jan 21, 2015 at 02:31:51PM +0100, Joanna Rutkowska wrote:
> > (...)
> >>
> >> Wouldn't it be prudent to also check if:
> >>
> >> (v->arch.paging.mode>{write_guest_entry,cmpxchg_guest_entry} != NULL)
> >>
> >> ... in the two affected functions, just before derefing these function
> >> pointers?
> >>
> >> Going even a step further: how about replacing all
> >> function-pointer-based calls with macros that first validates the
> >> pointer before derefing it? At least when the system doesn't have SMEP?
> >>
>
> ...to which I got a reply from one of the Xen Security Team engineers that
> the
> above might perhaps be justified in debug builds only, followed by a
> standard:
> "feel free to contribute a patch".
>
> b. The XSA-123 [6] was another critical security bug in Xen, this time
> resulting
> from one of the hypervisor developer's fetish to use an absolutely
> confusing
> construct in order to save a few modest bytes in a structure which might
> have
> been allocated by the system maybe a few tens of times at best. Even more
> worrying was the way how Xen Security Team decided to fix the bug: again by
> modifying some condition in the code further up the execution path, with
> the
> hopes that this time they would ensure this puzzling construct would
> always be
> used properly. We wrote more about this in our QSB #18 [7].
>
> c. Finally, the way how Xen fixed the recent XSA-148 looks also very
> reactive,
> IMHO. With a bug of this calibre, I would expect Xen to carefully review
> and
> augment all its PV memory virtualization code with additional checks
> (ASSERTs),
> ensuring certain invariants are always satisfied. Such as e.g. that none
> of the
> pages containing PDEs or PTEs are becoming writeable by the VM.
>
> I can't help but have a feeling that some of the Xen developers seem to be
> overconfident in their belief they can fully understand all the possible
> execution paths in their code. Well, the XSAs quoted above are an
> indisputable
> prove that this is not quite always the case. Realizing that, each
> developer by
> themselves, might be a great step towards a more secure hypervisor...
>
> 2. Another security-related aspect of the Xen project is how it totally
> ignores
> problems related to the build process security. Those who don't believe me
> should grep the sources for wget, which is now disguised as "FETCHER" shell
> variable... (so grep for "FETCHER" string)
>
> I feel embarrassed that I need to explain, at the end of 2015, why the
> build
> process of any serious software project should not blindly download
> unsigned
> components (sources) from the Internet, especially if it is about to
> execute
> Makefiles from these components a moment later... Come on, guys!
>
> (Of course we have been forced to get around this gapping security whole in
> Qubes OS [8] ourselves, sadly with a method that is not well suited for
> upstreaming).
>
> 3. Another thing is, of course: stop adding features to the core
> hypervisor. We
> really need Xen to finally mature, stabilize, and for its development
> process to
> be slowing down over time (just the bug fixes). We need a
> long-term-supported
> hypervisor, which doesn't change with subsonic speed. This would allow
> this core
> code to be widely audited by many experts. If some users want features,
> these
> should perhaps be maintained as additional modules (no, I don't mean
> dynamically
> loaded modules, just compile-time included), preferably in separate repos.
>
> Perhaps also to move all the non-hypervisor code, such as all the
> toolstacks,
> stubdom, etc, into separate repos also. For hygiene, if for nothing else.
>
> Admittedly, some of the features are a result of hardware evolution, such
> as
> e.g. UEFI support. But many are not. Again, maintaining these as optional
> code
> (in separate repos) would be a great step into getting the hypervisor
> maturing,
> finally.
>
> I have already written about it years ago [9], as a matter of fact.
>
> 4. Finally, I've been really surprised by the line of reasoning Ian
> expressed in
> the above-mentioned blog post. TL;DR: "we're still doing pretty great,
> compared
> to other projects, because: 1) we have smaller number of publicly disclosed
> bugs, and 2) we actually publicly disclose these bugs which we are aware
> of".
>
> The attitude presented in the blog post is so wrong, that I'm not even sure
> where to start commenting on this...
>
> With a single bug like the XSA-148 which, let me repeat that one more
> time: had
> been present in the hypervisor for the last 7 years, so with a bug like
> this it
> really doesn't matter how many (i.e. what number) of critical bugs does the
> competition have. Because only one bug of this calibre is enough for the
> attacker to never really bother to find another one. The mere fact that
> competing hypervisors might got 12 bugs during the same period, really
> doesn't
> make Xen look any better, sorry.
>
> Also, there is really nothing to be proud that you disclose the bugs. It
> would
> be a problem if you didn't.
>
> Hope the above comments might help improve the Xen security. Perhaps some
> would
> perceive them as arrogant or rude. Too bad. Remember the actual attackers
> will
> not be arrogant or rude -- they will just come and exploit bugs, silently.
> Admittedly this might not hurt some of the developers ego, not in the
> short time
> at least.
>
> Can we, the Qubes OS project, or myself personally, help with implementing
> the
> above suggestions? Sadly, no. While some of us do contribute occasional
> patches
> to Xen (specifically Marek Marczykowski-Górecki), we really work for a
> different
> project and have different tasks and responsibilities.
>
> Regards,
> joanna.
>
> [1] http://xenbits.xen.org/xsa/advisory-148.html
> [2]
> https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-022-2015.txt
> [3] https://blog.xenproject.org/2015/10/30/security-vs-features/
> [4] https://twitter.com/xen_org/status/660151720463482880
> [5] http://xenbits.xen.org/xsa/advisory-109.html
> [6] http://xenbits.xen.org/xsa/advisory-123.html
> [7]
> https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-018-2015.txt
> [8]
> https://github.com/QubesOS/qubes-vmm-xen/commit/dcd6c0a4f2c6226a9b706e62469d420579c86975
> [9] http://lists.xen.org/archives/html/xen-devel/2013-09/msg01815.html
> -----BEGIN PGP SIGNATURE-----
>
> iQIcBAEBAgAGBQJWPOHUAAoJEDOT2L8N3GcYzpEP/An/PTnKDOaC4tPKw3+Y8VIL
> n/xIfRPRnPRy18Tbx6EnrKzgohtVvtigtmd/FIxjYVuZ3Luw2B4RFSqUENg758Aa
> ANLs4kUD+yaUO82Jfg1nq/6PXBNZlKFovQuYV20LEW9JV6DvMCbzYJ2evZ6t0XS/
> EAhUOP1OqY4vb0kah4dwhQKepqwPcD5Tm5LLZn/qbO30e2zN9MkKB851vguQtVIz
> k5I8pv+MSQp1efRG2eg470onGtU36IIYFsY1OLihJA9MYh+74FpIA1xoURenJg6+
> NJhXEDnxxlz78BJaGOiSwHwB59yd2DXDJKAaUNV/H1LqQu3o1ED+8IZWUARkc0Wl
> ckTfQz/++exDhyRcWVHF5GnxEHWdyu/gNZOCNjl4o4HiYS4SQrhTRn7rWwalbyBB
> /jG3bAnU8m/Gtp95FtuWXCwuKeeOeBSfnxKMrksxu3JFSNevsYPZu5lrdtUEGLZA
> 97SwLj70GLesvMqEV3k7XmrQyt8LwyBzLCm+cCocaPEmOQAymeuslrs/RehjGSCQ
> L34Ipjvg85GoND64N8X56NuvD+/LrteRhp8hS+aEWv2YpNVJB9tmcOTzLzf7faPK
> KPyqa2lW7XKwm7n0WCbtPnrVHRbFPbKvkJRDnfPDwEAEiZcj0SjJ8h7fIDSQ4qac
> vgYBTJr/2cRrtC4y1An5
> =zQmw
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>
[-- Attachment #1.2: Type: text/html, Size: 10803 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [qubes-devel] Re: Critique of the Xen Security Process
2015-11-09 22:00 ` chris
@ 2015-11-10 2:46 ` Radoslaw Szkodzinski
2015-11-11 20:14 ` chris
0 siblings, 1 reply; 25+ messages in thread
From: Radoslaw Szkodzinski @ 2015-11-10 2:46 UTC (permalink / raw
To: xen-devel@lists.xen.org, qubes-devel@googlegroups.com
As usual. Security, performance, convenience, price. Pick any mixture.
As is usual for most software, developer convenience trumps most other
considerations. I include ease of generating nice papers and jobs
under developer convenience.
Big players are much more concerned about performance, which saves
money on machines you need to buy. (Note the resources poured into
tmem, which is very iffy from security standpoint.)
Even if there was a sudden security drive for Xen, it's a huge and
probably fruitless task. Xen is not designed top down so you get no
nice overview to check main assumptions. This is also why Xen is
written in C and assembly, not in anything easier to use.
Likewise security-oriented microkernels such as seL4 push drivers to
"userspace" making it convenient for their developers, at the cost of
performance and security, since they are now Somebody Else's Problem.
Surprisingly, seL4 seems to be the only active and working
non-research project that happens to have security as main aim, and
they don't even implement many of features that are "required" for
modern secure design, say, IOMMU...
They want you to write your drivers as automated proofs instead.
Meaning cheap developers will not be able to even lift a finger in
writing those, since you need to learn Isabelle DSL and understand
what is required in order to write a working driver with a good API,
then also understand and model features of underlying hardware. This
requires good mathematical background, as in actual CS, not just
writing code.
What seL4 folks are actually doing (have done, even) is writing a
potent compiler between both C and theorem proving language, plus a
set of theorems describing assumptions made on underlying hardware and
security features. Very simple, just some 250k lines of theorems to
prove correctness of 7k lines of C code.
Good luck trying to port Qubes to their architecture. It's not
impossible, but it is quite a task. So many theorems to wade through
and understand, not to mention a whole new programming language.
You'd get to add well defined isolation primitives to their theorem
prover, then write a parallel proof to capDL kernel (ouch) or extend
capDL with such isolation features and requisite APIs.
Again, developer convenience suffers and have too high barrier of
entry, and you won't get any code written or it will get too expensive
in terms of time, skill and therefore money.
As to other attempts...
Python code? Forget about it, Python is thoroughly undefined behavior
with no truly defined memory model. :) Even C++11 is better there.
Rust? Rust mechanisms are undefined in terms of timing, which will
result in unbounded execution time (hangs) when actual hardware is
involved - there is no way to describe timing behavior of hardware in
it in such a way that these properties are verified at higher layers.
They attempt concurrency but the primitives are not bound to actual
real time and hardware. There is no library for bare metal
programming. Memory allocations are assumed to never fail - and many
other things as well. (Does this sound like some other "never fails"
software we're depending upon?)
Trying to write a secure kernel in an unsafe language is akin to
trying to bail a sinking ship with a spoon. It can theoretically be
done, just not in practice. And actual low-level and safe languages
are surprisingly rare.
Best regards,
R.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [qubes-devel] Re: Critique of the Xen Security Process
2015-11-09 16:31 ` [qubes-devel] " Franz
2015-11-09 18:15 ` Wojtek Porczyk
@ 2015-11-10 10:52 ` Lars Kurth
2015-11-11 11:36 ` Chris Laprise
1 sibling, 1 reply; 25+ messages in thread
From: Lars Kurth @ 2015-11-10 10:52 UTC (permalink / raw
To: Franz
Cc: qubes-devel@googlegroups.com, Joanna Rutkowska, Jan Beulich,
xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 3637 bytes --]
Hi everyone,
firstly I wanted to thank everyone for raising this issue. I wanted to point out that we are not talking about a security process here, but the development process. Or more accurately the cost of writing more secure code and the relative importance of security compared to features. And of course the recent increase of relative importance of "built-in security as a feature" since Snowden.
> On 9 Nov 2015, at 16:31, Franz <169101@gmail.com> wrote:
>
> (Please note that all of the above are my personal views, i.e. I'm
> explicitly not speaking on behalf of the project.)
>
>
> It seems positions are antithetic with no possible compromise in view. Sadly this is a general problem of FOSS software: developers tend to do what they like and not what users request. And who can blame developers for that? After all they are working as volunteers so they deserve to do what they like. No possible compromise.
I don't think this is a fair statement: more than 95% of developers working on Xen are employees of large organisations. And they follow the priorities that their employers set. The same is true in Linux and for comparable projects and of course for proprietary software developers. Blaming open source developers, is simply wrong and not constructive.
In fact, throughout 2014 and 2015, the project has received complaints from several large vendors that the Xen Project today is too rigorous with code reviews compared to Linux, KVM and QEMU and that it is now harder to get code into Xen compared to similar projects. To understand the root causes, the Xen Project Advisory Board has commissioned a study to look at the root causes (see http://lists.xenproject.org/archives/html/xen-devel/2015-10/msg01635.html <http://lists.xenproject.org/archives/html/xen-devel/2015-10/msg01635.html>). The result of the first phase of the study showed, that a primary factor for longer review times are significantly higher quality expectations on new code compared to the past. Of course this does not help with code that has been added in the past.
In many ways, the problem which was stated comes down to different expectations, by different classes of contributors. And a solution, will necessarily have to involve compromise.
> Perhaps a way out of this impasse is to put bounties on Xen security tasks identified by Joanna and properly advertise these bounties to Xen users. I would give particular attention to the companies that use Xen.
I can certainly raise this suggestion with the Advisory Board and see whether we can make some funds available. However, the board has already invested nearly 50% of its entire budget in Test Infrastructure and is planning to continue to spend in this area at roughly this proportion of the projects budget. However, we do not have huge amounts of funds: thus, what we could do with bounties would necessarily be limited.
I personally also looked at other ways to change the cost-benefit equation. One example is the Feature Maturity Lifecycle (see http://lists.xen.org/archives/html/xen-devel/2015-11/msg00609.html <http://lists.xen.org/archives/html/xen-devel/2015-11/msg00609.html> & http://lists.xenproject.org/archives/html/xen-devel/2015-06/msg01992.html <http://lists.xenproject.org/archives/html/xen-devel/2015-06/msg01992.html>) which aims to use better classification to change behaviour of contributors.
I do hope, that this discussion can remain constructive. De-motivating the good work many of our developers (and in particular code reviewers) are doing, is really not helpful in this context.
Best Regards
Lars
[-- Attachment #1.2: Type: text/html, Size: 6131 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [qubes-devel] Re: Critique of the Xen Security Process
2015-11-09 18:15 ` Wojtek Porczyk
@ 2015-11-10 13:09 ` Lars Kurth
2015-11-10 14:10 ` Franz
0 siblings, 1 reply; 25+ messages in thread
From: Lars Kurth @ 2015-11-10 13:09 UTC (permalink / raw
To: Wojtek Porczyk
Cc: Franz, qubes-devel@googlegroups.com, Jan Beulich,
Joanna Rutkowska, xen-devel
> On 9 Nov 2015, at 18:15, Wojtek Porczyk <woju@invisiblethingslab.com> wrote:
>
> On Mon, Nov 09, 2015 at 04:31:58PM +0000, Franz wrote:
>> Perhaps a way out of this impasse is to put bounties on Xen security tasks
>> identified by Joanna and properly advertise these bounties to Xen users.
>> [snip]
>
> This is fundamentaly wrong idea. Security isn't something you can
> "apply" or put bounty on. It's a state of the mind, especcialy
> developer's.
I don't think this was what was proposed. What was proposed, if I understood correctly, was to use funds to sponsor solutions to specific problems in specific areas of code that have been developed in the past and where we may have issues due to past mistakes.
> Joanna wrote in her mail:
>
>>>> I can't help but have a feeling that some of the Xen developers seem to be
>>>> overconfident in their belief they can fully understand all the possible
>>>> execution paths in their code. Well, the XSAs quoted above are an indisputable
>>>> prove that this is not quite always the case. Realizing that, each developer by
>>>> themselves, might be a great step towards a more secure hypervisor...
>
> And that's why we can't just "submit a patch" to "contribute security".
Although I agree that you can't submit a patch to fix security, it is possible to focus on some areas you care about and lead by example. In particular, if you believe that there is a mindset problem today. I think, you do also need to distinguish between what was happening in the past and what the project does today. There is little point to complain about what developers, who now don't work on Xen any more, did in the past. In particular, as the vast majority of active developers in the project were not around 6-7 years ago.
> There is something wrong with Xen as a whole project, but that something
> isn't the code. There is a mindset to be fixed.
I think this is a rather generalising statement, and maybe it would be more fruitful to focus on some concrete areas that can be improved. I also believe, that there has been a significant shift in mindset in many areas in the project in the last few years.
Lars
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [qubes-devel] Re: Critique of the Xen Security Process
2015-11-10 13:09 ` Lars Kurth
@ 2015-11-10 14:10 ` Franz
0 siblings, 0 replies; 25+ messages in thread
From: Franz @ 2015-11-10 14:10 UTC (permalink / raw
To: Lars Kurth
Cc: Wojtek Porczyk, qubes-devel@googlegroups.com, Jan Beulich,
Joanna Rutkowska, xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 3327 bytes --]
On Tue, Nov 10, 2015 at 1:09 PM, Lars Kurth <lars.kurth.xen@gmail.com>
wrote:
>
> > On 9 Nov 2015, at 18:15, Wojtek Porczyk <woju@invisiblethingslab.com>
> wrote:
> >
> > On Mon, Nov 09, 2015 at 04:31:58PM +0000, Franz wrote:
> >> Perhaps a way out of this impasse is to put bounties on Xen security
> tasks
> >> identified by Joanna and properly advertise these bounties to Xen users.
> >> [snip]
> >
> > This is fundamentaly wrong idea. Security isn't something you can
> > "apply" or put bounty on. It's a state of the mind, especcialy
> > developer's.
>
> I don't think this was what was proposed. What was proposed, if I
> understood correctly, was to use funds to sponsor solutions to specific
> problems in specific areas of code that have been developed in the past and
> where we may have issues due to past mistakes.
>
>
Yes Lars of course, and also changing mindset has real costs, because a
more security oriented mindset involves some additional work, some hard
thinking, some additional study, something already done that needs recoding
etc. I mean it is not simply an act of will, rather a targeted effort. A
security mindset is slowing developing in the whole society and I see Qubes
developers as the more advanced front. The reality is changing fast. A
serious bug in Xen practically means that all my passwords can be stolen
from my no-network VM or even more seriously that the life of an activist
can be compromised. These facts are dramatic or tragic and the situation is
constantly and fast worsening. So nobody is guilty for 7 years old code
developed in a different situation, but the same Joanna words are totally
justified and welcome. We have to face reality and take action. From my
point of view It is a matter of educating those who pay, for that I offered
to write some copywriting.
Best
Fran
> > Joanna wrote in her mail:
> >
> >>>> I can't help but have a feeling that some of the Xen developers seem
> to be
> >>>> overconfident in their belief they can fully understand all the
> possible
> >>>> execution paths in their code. Well, the XSAs quoted above are an
> indisputable
> >>>> prove that this is not quite always the case. Realizing that, each
> developer by
> >>>> themselves, might be a great step towards a more secure hypervisor...
> >
> > And that's why we can't just "submit a patch" to "contribute security".
>
> Although I agree that you can't submit a patch to fix security, it is
> possible to focus on some areas you care about and lead by example. In
> particular, if you believe that there is a mindset problem today. I think,
> you do also need to distinguish between what was happening in the past and
> what the project does today. There is little point to complain about what
> developers, who now don't work on Xen any more, did in the past. In
> particular, as the vast majority of active developers in the project were
> not around 6-7 years ago.
>
> > There is something wrong with Xen as a whole project, but that something
> > isn't the code. There is a mindset to be fixed.
>
> I think this is a rather generalising statement, and maybe it would be
> more fruitful to focus on some concrete areas that can be improved. I also
> believe, that there has been a significant shift in mindset in many areas
> in the project in the last few years.
>
> Lars
>
>
[-- Attachment #1.2: Type: text/html, Size: 4273 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Critique of the Xen Security Process
2015-11-09 21:48 ` Doug Goldstein
@ 2015-11-11 9:43 ` Ian Campbell
2015-11-11 9:59 ` Lars Kurth
2015-11-11 12:33 ` Raisin, was " Stefano Stabellini
2015-11-11 12:34 ` Wei Liu
1 sibling, 2 replies; 25+ messages in thread
From: Ian Campbell @ 2015-11-11 9:43 UTC (permalink / raw
To: Doug Goldstein, xen-devel
On Mon, 2015-11-09 at 15:48 -0600, Doug Goldstein wrote:
>
> I'll echo this sentiment as well. Most distro packagers will dislike
> this and need to work around some of this behavior in their respective
> distros.
This is something we have been working upstream to address as well. As it
stands I believe everything which the tools might download can be
redirected to instead an existing component (via one of the --with-system-
foo configuration options) or disabled (via a --disable-foo configure
option). So I think now the current state is that there aren't
"workarounds" but rather "supported ways to disable".
The big outstanding issue is the stubdom build, the distro I care about
most (Debian) simply doesn't build these (for reasons above and beyond the
downloading).
> Project Raisin is aiming to help with this
Indeed, and it might also allow us to make some of the above options the
default in the future.
Maybe in the meantime perhaps a ./configure --ensure-offline or --disable-
downloads which:
* either disables stubdoms automatically or checks you've passed --
disable-stubdom as well
* either disables all the other things which might be cloned or requires
the corresponding --with-system-foo=, or has a guess at a default system
version
* sets FETCHER to /bin/false
would be useful? (essentially as a guard against new options being required
to turn stuff off).
> but it doesn't seem
> to have a lot of community effort behind it and it too attempts to
> install dependencies on my machine and wants to be run with sudo.
I believe it has a mode where it simply checks for dependencies and tells
you what is required and thereby avoids the need for sudo, but I'm not
sure.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Critique of the Xen Security Process
2015-11-11 9:43 ` Ian Campbell
@ 2015-11-11 9:59 ` Lars Kurth
2015-11-11 17:21 ` Lars Kurth
2015-11-11 12:33 ` Raisin, was " Stefano Stabellini
1 sibling, 1 reply; 25+ messages in thread
From: Lars Kurth @ 2015-11-11 9:59 UTC (permalink / raw
To: Ian Campbell
Cc: qubes-devel, James Bulpin, Doug Goldstein, Joanna Rutkowska,
Xen-devel
> On 11 Nov 2015, at 09:43, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>
>> Project Raisin is aiming to help with this
>
> Indeed, and it might also allow us to make some of the above options the
> default in the future.
>
> Maybe in the meantime perhaps a ./configure --ensure-offline or --disable-
> downloads which:
> * either disables stubdoms automatically or checks you've passed --
> disable-stubdom as well
> * either disables all the other things which might be cloned or requires
> the corresponding --with-system-foo=, or has a guess at a default system
> version
> * sets FETCHER to /bin/false
>
> would be useful? (essentially as a guard against new options being required
> to turn stuff off).
>
>> but it doesn't seem
>> to have a lot of community effort behind it and it too attempts to
>> install dependencies on my machine and wants to be run with sudo.
>
> I believe it has a mode where it simply checks for dependencies and tells
> you what is required and thereby avoids the need for sudo, but I'm not
> sure.
It seems that raisin may provide a good baseline the for "build process security", but it would of course be good to hear this from others who have raised this issue. Assuming it is (we probably need a few ACKs for this), would it make sense to take this into a separate thread then (with an appropriate CC list), and refer to it from here?
Regards
Lars
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [qubes-devel] Re: Critique of the Xen Security Process
2015-11-10 10:52 ` Lars Kurth
@ 2015-11-11 11:36 ` Chris Laprise
2015-11-11 15:24 ` Lars Kurth
0 siblings, 1 reply; 25+ messages in thread
From: Chris Laprise @ 2015-11-11 11:36 UTC (permalink / raw
To: Lars Kurth, Franz
Cc: qubes-devel@googlegroups.com, Joanna Rutkowska, Jan Beulich,
xen-devel
Hello...
On 11/10/2015 05:52 AM, Lars Kurth wrote:
> Hi everyone,
>
> firstly I wanted to thank everyone for raising this issue. I wanted to
> point out that we are not talking about a security process here, but
> the development process. Or more accurately the cost of writing more
> secure code and the relative importance of security compared to
> features. And of course the recent increase of relative importance of
> "built-in security as a feature" since Snowden.
>
>> On 9 Nov 2015, at 16:31, Franz <169101@gmail.com
>> <mailto:169101@gmail.com>> wrote:
>>
>> (Please note that all of the above are my personal views, i.e. I'm
>> explicitly not speaking on behalf of the project.)
>>
>>
>> It seems positions are antithetic with no possible compromise in
>> view. Sadly this is a general problem of FOSS software: developers
>> tend to do what they like and not what users request. And who can
>> blame developers for that? After all they are working as volunteers
>> so they deserve to do what they like. No possible compromise.
>
> I don't think this is a fair statement: more than 95% of developers
> working on Xen are employees of large organisations. And they follow
> the priorities that their employers set. The same is true in Linux and
> for comparable projects and of course for proprietary software
> developers. Blaming open source developers, is simply wrong and not
> constructive.
Nevertheless, Xen is a creature of interfaces that purport to uphold
contracts. These (software engineering, not legal) contracts are
explicitly security-themed... the software is promising to isolate
processes from each other and protect against privilege escalation. The
Xen project itself asserts that it is focused on the security benefits
of the hypervisor (to the point of invoking "microkernel" in Xen's
description), not mere administrative convenience.
What seems to be missing from the defense of Xen project so far in this
thread is a level of acknowledgement of this very important aspect of
twenty-first century software development: Can you honor what your
interfaces communicate to your audience? Its true that some FOSS
projects do not care for modern methodologies (which are greatly about
concept and mindset), but should Xen be that way?
Should Xen project also be a blanket absolution of "blame"? And is the
corporate status of some of its contributors a proper justification for
being blame-less (perhaps we can think of them like members of 'The
Borg')? I think not.
This is the trap that FOSS projects most commonly fall into: 'We are
free to publish because of liberty, but you are not free to criticize.'
Its odd when you think about it. This is the squandering of user
motivation and valuable feedback.
>
> In fact, throughout 2014 and 2015, the project has received complaints
> from several large vendors that the Xen Project today is too rigorous
> with code reviews compared to Linux, KVM...
Famous last words before a Heartbleed-scale media circus ensues?
>
> I can certainly raise this suggestion with the Advisory Board and see
> whether we can make some funds available. However, the board has
> already invested nearly 50% of its entire budget in Test
> Infrastructure and is planning to continue to spend in this area at
> roughly this proportion of the projects budget. However, we do not
> have huge amounts of funds: thus, what we could do with bounties would
> necessarily be limited.
>
> I personally also looked at other ways to change the cost-benefit
> equation. One example is the Feature Maturity Lifecycle (see
> http://lists.xen.org/archives/html/xen-devel/2015-11/msg00609.html &
> http://lists.xenproject.org/archives/html/xen-devel/2015-06/msg01992.html)
> which aims to use better classification to change behaviour of
> contributors.
>
> I do hope, that this discussion can remain constructive. De-motivating
> the good work many of our developers (and in particular code
> reviewers) are doing, is really not helpful in this context.
>
It does seem to me that the suggestion of a Long-Term Support (LTS)
release is a constructive one, among the other suggestions that Joanna
made. The FML you cite is interesting, but seems to be aimed squarely at
developers. You are bound to get better results if the expectations of
users change along with developers, so LTS releases may be the better idea.
> Best Regards
> Lars
>
>
^ permalink raw reply [flat|nested] 25+ messages in thread
* Raisin, was Critique of the Xen Security Process
2015-11-11 9:43 ` Ian Campbell
2015-11-11 9:59 ` Lars Kurth
@ 2015-11-11 12:33 ` Stefano Stabellini
2015-11-11 16:24 ` Doug Goldstein
1 sibling, 1 reply; 25+ messages in thread
From: Stefano Stabellini @ 2015-11-11 12:33 UTC (permalink / raw
To: Ian Campbell; +Cc: Doug Goldstein, xen-devel
[-- Attachment #1: Type: text/plain, Size: 2234 bytes --]
On Wed, 11 Nov 2015, Ian Campbell wrote:
> On Mon, 2015-11-09 at 15:48 -0600, Doug Goldstein wrote:
> >
> > I'll echo this sentiment as well. Most distro packagers will dislike
> > this and need to work around some of this behavior in their respective
> > distros.
>
> This is something we have been working upstream to address as well. As it
> stands I believe everything which the tools might download can be
> redirected to instead an existing component (via one of the --with-system-
> foo configuration options) or disabled (via a --disable-foo configure
> option). So I think now the current state is that there aren't
> "workarounds" but rather "supported ways to disable".
>
> The big outstanding issue is the stubdom build, the distro I care about
> most (Debian) simply doesn't build these (for reasons above and beyond the
> downloading).
Yes indeed. I have been tempted to disable stubdoms in Raisin until they
are properly integrated in it.
> > Project Raisin is aiming to help with this
>
> Indeed, and it might also allow us to make some of the above options the
> default in the future.
>
> Maybe in the meantime perhaps a ./configure --ensure-offline or --disable-
> downloads which:
> * either disables stubdoms automatically or checks you've passed --
> disable-stubdom as well
> * either disables all the other things which might be cloned or requires
> the corresponding --with-system-foo=, or has a guess at a default system
> version
> * sets FETCHER to /bin/false
>
> would be useful? (essentially as a guard against new options being required
> to turn stuff off).
>
> > but it doesn't seem
> > to have a lot of community effort behind it and it too attempts to
> > install dependencies on my machine and wants to be run with sudo.
>
> I believe it has a mode where it simply checks for dependencies and tells
> you what is required and thereby avoids the need for sudo, but I'm not
> sure.
Yes, that is correct. Raisin won't try to use sudo before asking the
user first. That is the expected behaviour, if it doesn't work that way
is a bug.
Moreover I would be happy to introduce signature checks on git clones
and downloads in Raisin.
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Critique of the Xen Security Process
2015-11-09 21:48 ` Doug Goldstein
2015-11-11 9:43 ` Ian Campbell
@ 2015-11-11 12:34 ` Wei Liu
1 sibling, 0 replies; 25+ messages in thread
From: Wei Liu @ 2015-11-11 12:34 UTC (permalink / raw
To: Doug Goldstein; +Cc: wei.liu2, xen-devel
On Mon, Nov 09, 2015 at 03:48:42PM -0600, Doug Goldstein wrote:
[...]
> As far as the compile time support goes I'm aiming for this as well.
> I've been working on Kconfig support and hope to get that pushed soon
> and the idea being that less mature features can be kept off until
> they're ready to be default on. The downstream distro/product makers can
> then fiddle with these flags to get the hypervisor that fits their needs.
>
> > Perhaps also to move all the non-hypervisor code, such as all the toolstacks,
> > stubdom, etc, into separate repos also. For hygiene, if for nothing else.
>
> From a real high level things should be broken into at least 4 repos:
> 1) hypervisor
> 2) libxl
> 3) userland tools (xl and support scripts)
> 4) stubdom
Stubdom can be a starter. It would be trivial to teach Xen build system
to pull from a repo somewhere. But then the released tarball will still
contain stubdom source code. I think most distros care about building
from tarball? If what you're thinking is to make stubdom able to do
standalone build so that it doesn't show up in final tarball or xen.git,
that would require a bit of work.
Can you clarify your use case?
Wei.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Critique of the Xen Security Process
2015-11-06 17:22 Critique of the Xen Security Process Joanna Rutkowska
` (5 preceding siblings ...)
2015-11-09 22:00 ` chris
@ 2015-11-11 12:59 ` Stefano Stabellini
6 siblings, 0 replies; 25+ messages in thread
From: Stefano Stabellini @ 2015-11-11 12:59 UTC (permalink / raw
To: Joanna Rutkowska; +Cc: qubes-devel, xen-devel
[-- Attachment #1: Type: text/plain, Size: 1816 bytes --]
On Fri, 6 Nov 2015, Joanna Rutkowska wrote:
> Can we, the Qubes OS project, or myself personally, help with implementing the
> above suggestions? Sadly, no. While some of us do contribute occasional patches
> to Xen (specifically Marek Marczykowski-Górecki), we really work for a different
> project and have different tasks and responsibilities.
Ian wrote in his blog post: "Ultimately, of course, a Free Software
project like Xen is what the whole community makes it. In the project as
a whole we get a lot more submissions of new functionality than we get
submissions aimed at improving the security".
If invisiblethingslab, or other security groups, were among of the top
three contributors or reviewers in Xen Project, the community and the
code would be different.
If somebody submitted a clean patch to disable or remove a functionality
because it is not secure, we would consider it. Even better, if somebody
reviewed a patch and found security issues with it, she would prevent
the patch from getting in in the first place. What doesn't this happen
more often?
In general, if security is important to many people, why don't we see
more security oriented submissions and reviews? Anybody can do reviews!
He or she doesn't need to be a maintainer to do them.
Open Source software communities are a funnel:
https://talesfromthecommunity.wordpress.com/2012/06/16/viewing-communities-as-funnels/
and in particular:
https://talesfromthecommunity.files.wordpress.com/2012/06/basicfunnel1.png
It seems to me that not enough security aware people are getting past
the "customize without contributing back" or "occasionally contributing"
stages. Why is that?
Don't underestimate the impact that even a single very committed
individual can have on a software project.
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [qubes-devel] Re: Critique of the Xen Security Process
2015-11-11 11:36 ` Chris Laprise
@ 2015-11-11 15:24 ` Lars Kurth
0 siblings, 0 replies; 25+ messages in thread
From: Lars Kurth @ 2015-11-11 15:24 UTC (permalink / raw
To: Chris Laprise
Cc: Franz, qubes-devel@googlegroups.com, Joanna Rutkowska,
Jan Beulich, Xen-devel
> On 11 Nov 2015, at 11:36, Chris Laprise <tasket@openmailbox.org> wrote:
>
> Hello...
>
> On 11/10/2015 05:52 AM, Lars Kurth wrote:
>> Hi everyone,
>>
>> firstly I wanted to thank everyone for raising this issue. I wanted to point out that we are not talking about a security process here, but the development process. Or more accurately the cost of writing more secure code and the relative importance of security compared to features. And of course the recent increase of relative importance of "built-in security as a feature" since Snowden.
>>
>>> On 9 Nov 2015, at 16:31, Franz <169101@gmail.com<mailto:169101@gmail.com>> wrote:
>>>
>>> (Please note that all of the above are my personal views, i.e. I'm
>>> explicitly not speaking on behalf of the project.)
>>>
>>>
>>> It seems positions are antithetic with no possible compromise in view. Sadly this is a general problem of FOSS software: developers tend to do what they like and not what users request. And who can blame developers for that? After all they are working as volunteers so they deserve to do what they like. No possible compromise.
>>
>> I don't think this is a fair statement: more than 95% of developers working on Xen are employees of large organisations. And they follow the priorities that their employers set. The same is true in Linux and for comparable projects and of course for proprietary software developers. Blaming open source developers, is simply wrong and not constructive.
>
> Nevertheless, Xen is a creature of interfaces that purport to uphold contracts. These (software engineering, not legal) contracts are explicitly security-themed... the software is promising to isolate processes from each other and protect against privilege escalation. The Xen project itself asserts that it is focused on the security benefits of the hypervisor (to the point of invoking "microkernel" in Xen's description), not mere administrative convenience.
All of this is true from an architectural viewpoint and I don't think that has been disputed. And it also related to how we handle security issues, which we have proven to handle more effectively than most other projects and even commercial vendors.
> What seems to be missing from the defense of Xen project so far in this thread is a level of acknowledgement of this very important aspect of twenty-first century software development: Can you honor what your interfaces communicate to your audience? Its true that some FOSS projects do not care for modern methodologies (which are greatly about concept and mindset), but should Xen be that way?
There has been some discussion around this in the past. The problem is that the cost of doing this retrospectively (at least in the context of certification, which would be one way to achieve this) is prohibitively expensive. And the cost of doing this for new code, is also very high. You can find some cost calculations in: http://www.xenproject.org/presentations-and-videos/video/xpds14v-certification.html
You seem to be arguing that Xen Project should be like a security hardened Linux distro. The project never claimed that this was the case. Moving towards something better, may be achievable through some better working practice. But not without significant investment, which will affect other members of the community.
The fundamental questions is, whether for the majority of Xen users, what we do today is good enough or not. And whether we do better than competing projects and products. And I believe that compared to others we are doing better. Maybe the solution is for some of the security conscious vendors and users of Xen to collaborate and create a security hardened version of Xen, or to work with the community in other ways, rather than to ask others to pay them a free lunch.
> Should Xen project also be a blanket absolution of "blame"? And is the corporate status of some of its contributors a proper justification for being blame-less (perhaps we can think of them like members of 'The Borg')? I think not.
That was not what I argued against: what I argued against the statement that "this is a general problem of FOSS software: developers tend to do what they like and not what users request". Of course vendors who contribute to open source do so primarily because their customers are asking for features and other functionality. And IMHO, the pressure to constantly deliver new things has actually become worse in the last 2 years. You just need to look at the Container and Docker hype, to see the majority of the industry appear to value features and the latest "shiny new thing" a lot more than security (or slower development velocity as a consequence of security).
> This is the trap that FOSS projects most commonly fall into: 'We are free to publish because of liberty, but you are not free to criticize.' Its odd when you think about it. This is the squandering of user motivation and valuable feedback.
>
>
>>
>> In fact, throughout 2014 and 2015, the project has received complaints from several large vendors that the Xen Project today is too rigorous with code reviews compared to Linux, KVM...
>
>
> Famous last words before a Heartbleed-scale media circus ensues?
This is already happening: see http://www.washingtonpost.com/sf/business/2015/11/05/net-of-insecurity-the-kernel-of-the-argument/
>> I can certainly raise this suggestion with the Advisory Board and see whether we can make some funds available. However, the board has already invested nearly 50% of its entire budget in Test Infrastructure and is planning to continue to spend in this area at roughly this proportion of the projects budget. However, we do not have huge amounts of funds: thus, what we could do with bounties would necessarily be limited.
>>
>> I personally also looked at other ways to change the cost-benefit equation. One example is the Feature Maturity Lifecycle (see http://lists.xen.org/archives/html/xen-devel/2015-11/msg00609.html & http://lists.xenproject.org/archives/html/xen-devel/2015-06/msg01992.html) which aims to use better classification to change behaviour of contributors.
>>
>> I do hope, that this discussion can remain constructive. De-motivating the good work many of our developers (and in particular code reviewers) are doing, is really not helpful in this context.
>>
>
> It does seem to me that the suggestion of a Long-Term Support (LTS) release is a constructive one, among the other suggestions that Joanna made. The FML you cite is interesting, but seems to be aimed squarely at developers. You are bound to get better results if the expectations of users change along with developers, so LTS releases may be the better idea.
The FML is clearly targeted at developers - actually more at vendors who contribute to the project.
As for a LTS release, I assume what you meant is to allow security conscious people to have a long-term baseline of Xen, which does not include new features. We do have mechanisms for this in the community today: any vendor, individual or group of vendors are very welcome to own and maintain such a LTS release with support and backing from the project. In fact, this is not that hard and does not require that much effort. And I would also be willing to support such a proposal if some people stepped up and see whether more interested parties can be pulled in. To create a security hardened LTS branch however - which it seems you and others are arguing for - would of course be significantly more expensive.
That is not to say that we are unwilling to improve: I believe the project has a track record of improving working practices. But in the end trade-offs have to be made somewhere. And a realistic compromise will most likely fall short of a security hardened Xen LTS.
I don't think it is acceptable though to ask others to foot the bill for it, without contributing something. In particular if what we do today is good enough for those who put the effort in. That approach very much feels like "getting a free lunch and complaining that it isn't what you were hoping for".
Regards
Lars
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Raisin, was Critique of the Xen Security Process
2015-11-11 12:33 ` Raisin, was " Stefano Stabellini
@ 2015-11-11 16:24 ` Doug Goldstein
2015-11-11 17:40 ` George Dunlap
0 siblings, 1 reply; 25+ messages in thread
From: Doug Goldstein @ 2015-11-11 16:24 UTC (permalink / raw
To: Stefano Stabellini, Ian Campbell; +Cc: xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 2690 bytes --]
On 11/11/15 6:33 AM, Stefano Stabellini wrote:
> On Wed, 11 Nov 2015, Ian Campbell wrote:
>> On Mon, 2015-11-09 at 15:48 -0600, Doug Goldstein wrote:
>>>
>>> I'll echo this sentiment as well. Most distro packagers will dislike
>>> this and need to work around some of this behavior in their respective
>>> distros.
>>
>> This is something we have been working upstream to address as well. As it
>> stands I believe everything which the tools might download can be
>> redirected to instead an existing component (via one of the --with-system-
>> foo configuration options) or disabled (via a --disable-foo configure
>> option). So I think now the current state is that there aren't
>> "workarounds" but rather "supported ways to disable".
>>
>> The big outstanding issue is the stubdom build, the distro I care about
>> most (Debian) simply doesn't build these (for reasons above and beyond the
>> downloading).
>
> Yes indeed. I have been tempted to disable stubdoms in Raisin until they
> are properly integrated in it.
Define "properly integrated in". What work needs to be done to support
this? Until the tooling really makes it easy to use stubdoms then people
won't really use them and improve them.
>
>
>>> Project Raisin is aiming to help with this
>>
>> Indeed, and it might also allow us to make some of the above options the
>> default in the future.
>>
>> Maybe in the meantime perhaps a ./configure --ensure-offline or --disable-
>> downloads which:
>> * either disables stubdoms automatically or checks you've passed --
>> disable-stubdom as well
>> * either disables all the other things which might be cloned or requires
>> the corresponding --with-system-foo=, or has a guess at a default system
>> version
>> * sets FETCHER to /bin/false
>>
>> would be useful? (essentially as a guard against new options being required
>> to turn stuff off).
>>
>>> but it doesn't seem
>>> to have a lot of community effort behind it and it too attempts to
>>> install dependencies on my machine and wants to be run with sudo.
>>
>> I believe it has a mode where it simply checks for dependencies and tells
>> you what is required and thereby avoids the need for sudo, but I'm not
>> sure.
>
> Yes, that is correct. Raisin won't try to use sudo before asking the
> user first. That is the expected behaviour, if it doesn't work that way
> is a bug.
>
> Moreover I would be happy to introduce signature checks on git clones
> and downloads in Raisin.
>
Does it have the mode Ian mentions where it will just print the depends
out to the user instead of installing them for you?
--
Doug Goldstein
[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 959 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Critique of the Xen Security Process
2015-11-11 9:59 ` Lars Kurth
@ 2015-11-11 17:21 ` Lars Kurth
0 siblings, 0 replies; 25+ messages in thread
From: Lars Kurth @ 2015-11-11 17:21 UTC (permalink / raw
To: Ian Campbell
Cc: qubes-devel, James Bulpin, Doug Goldstein, Joanna Rutkowska,
Xen-devel
> On 11 Nov 2015, at 09:59, Lars Kurth <lars.kurth.xen@gmail.com> wrote:
>
>>> but it doesn't seem
>>> to have a lot of community effort behind it and it too attempts to
>>> install dependencies on my machine and wants to be run with sudo.
>>
>> I believe it has a mode where it simply checks for dependencies and tells
>> you what is required and thereby avoids the need for sudo, but I'm not
>> sure.
>
> It seems that raisin may provide a good baseline the for "build process security", but it would of course be good to hear this from others who have raised this issue. Assuming it is (we probably need a few ACKs for this), would it make sense to take this into a separate thread then (with an appropriate CC list), and refer to it from here?
The details of this are discussed at http://lists.xenproject.org/archives/html/xen-devel/2015-11/msg01045.html now
Lars
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Raisin, was Critique of the Xen Security Process
2015-11-11 16:24 ` Doug Goldstein
@ 2015-11-11 17:40 ` George Dunlap
2015-11-11 17:49 ` Stefano Stabellini
0 siblings, 1 reply; 25+ messages in thread
From: George Dunlap @ 2015-11-11 17:40 UTC (permalink / raw
To: Doug Goldstein; +Cc: xen-devel@lists.xen.org, Ian Campbell, Stefano Stabellini
On Wed, Nov 11, 2015 at 4:24 PM, Doug Goldstein <cardoe@cardoe.com> wrote:
> On 11/11/15 6:33 AM, Stefano Stabellini wrote:
>> On Wed, 11 Nov 2015, Ian Campbell wrote:
>>> On Mon, 2015-11-09 at 15:48 -0600, Doug Goldstein wrote:
>>>>
>>>> I'll echo this sentiment as well. Most distro packagers will dislike
>>>> this and need to work around some of this behavior in their respective
>>>> distros.
>>>
>>> This is something we have been working upstream to address as well. As it
>>> stands I believe everything which the tools might download can be
>>> redirected to instead an existing component (via one of the --with-system-
>>> foo configuration options) or disabled (via a --disable-foo configure
>>> option). So I think now the current state is that there aren't
>>> "workarounds" but rather "supported ways to disable".
>>>
>>> The big outstanding issue is the stubdom build, the distro I care about
>>> most (Debian) simply doesn't build these (for reasons above and beyond the
>>> downloading).
>>
>> Yes indeed. I have been tempted to disable stubdoms in Raisin until they
>> are properly integrated in it.
>
> Define "properly integrated in". What work needs to be done to support
> this? Until the tooling really makes it easy to use stubdoms then people
> won't really use them and improve them.
I think Stefano is talking about integrating stubdoms as an external
build in Raisin. At the moment, I think stubdoms are one of the only
things that will still download random external tarballs when you're
building with raisin.
>
>>
>>
>>>> Project Raisin is aiming to help with this
>>>
>>> Indeed, and it might also allow us to make some of the above options the
>>> default in the future.
>>>
>>> Maybe in the meantime perhaps a ./configure --ensure-offline or --disable-
>>> downloads which:
>>> * either disables stubdoms automatically or checks you've passed --
>>> disable-stubdom as well
>>> * either disables all the other things which might be cloned or requires
>>> the corresponding --with-system-foo=, or has a guess at a default system
>>> version
>>> * sets FETCHER to /bin/false
>>>
>>> would be useful? (essentially as a guard against new options being required
>>> to turn stuff off).
>>>
>>>> but it doesn't seem
>>>> to have a lot of community effort behind it and it too attempts to
>>>> install dependencies on my machine and wants to be run with sudo.
>>>
>>> I believe it has a mode where it simply checks for dependencies and tells
>>> you what is required and thereby avoids the need for sudo, but I'm not
>>> sure.
>>
>> Yes, that is correct. Raisin won't try to use sudo before asking the
>> user first. That is the expected behaviour, if it doesn't work that way
>> is a bug.
>>
>> Moreover I would be happy to introduce signature checks on git clones
>> and downloads in Raisin.
>>
>
> Does it have the mode Ian mentions where it will just print the depends
> out to the user instead of installing them for you?
Not exactly. At the moment it will determine, based on your OS and
what you're trying to compile, which packages you need, and of those
which packages are not installed, to generate a list of packages to
install. If you pass "-v", and require a password for sudo, it will
print the list it is about to install before prompting you for the
password, at which point you could copy & paste if you want to.
So all the functionality is there internally, it's just a matter of
adding commands to make it only print the list rather than trying to
actually install them.
-George
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Raisin, was Critique of the Xen Security Process
2015-11-11 17:40 ` George Dunlap
@ 2015-11-11 17:49 ` Stefano Stabellini
0 siblings, 0 replies; 25+ messages in thread
From: Stefano Stabellini @ 2015-11-11 17:49 UTC (permalink / raw
To: George Dunlap
Cc: xen-devel@lists.xen.org, Doug Goldstein, Ian Campbell,
Stefano Stabellini
On Wed, 11 Nov 2015, George Dunlap wrote:
> On Wed, Nov 11, 2015 at 4:24 PM, Doug Goldstein <cardoe@cardoe.com> wrote:
> > On 11/11/15 6:33 AM, Stefano Stabellini wrote:
> >> On Wed, 11 Nov 2015, Ian Campbell wrote:
> >>> On Mon, 2015-11-09 at 15:48 -0600, Doug Goldstein wrote:
> >>>>
> >>>> I'll echo this sentiment as well. Most distro packagers will dislike
> >>>> this and need to work around some of this behavior in their respective
> >>>> distros.
> >>>
> >>> This is something we have been working upstream to address as well. As it
> >>> stands I believe everything which the tools might download can be
> >>> redirected to instead an existing component (via one of the --with-system-
> >>> foo configuration options) or disabled (via a --disable-foo configure
> >>> option). So I think now the current state is that there aren't
> >>> "workarounds" but rather "supported ways to disable".
> >>>
> >>> The big outstanding issue is the stubdom build, the distro I care about
> >>> most (Debian) simply doesn't build these (for reasons above and beyond the
> >>> downloading).
> >>
> >> Yes indeed. I have been tempted to disable stubdoms in Raisin until they
> >> are properly integrated in it.
> >
> > Define "properly integrated in". What work needs to be done to support
> > this? Until the tooling really makes it easy to use stubdoms then people
> > won't really use them and improve them.
>
> I think Stefano is talking about integrating stubdoms as an external
> build in Raisin. At the moment, I think stubdoms are one of the only
> things that will still download random external tarballs when you're
> building with raisin.
Right
> >>
> >>
> >>>> Project Raisin is aiming to help with this
> >>>
> >>> Indeed, and it might also allow us to make some of the above options the
> >>> default in the future.
> >>>
> >>> Maybe in the meantime perhaps a ./configure --ensure-offline or --disable-
> >>> downloads which:
> >>> * either disables stubdoms automatically or checks you've passed --
> >>> disable-stubdom as well
> >>> * either disables all the other things which might be cloned or requires
> >>> the corresponding --with-system-foo=, or has a guess at a default system
> >>> version
> >>> * sets FETCHER to /bin/false
> >>>
> >>> would be useful? (essentially as a guard against new options being required
> >>> to turn stuff off).
> >>>
> >>>> but it doesn't seem
> >>>> to have a lot of community effort behind it and it too attempts to
> >>>> install dependencies on my machine and wants to be run with sudo.
> >>>
> >>> I believe it has a mode where it simply checks for dependencies and tells
> >>> you what is required and thereby avoids the need for sudo, but I'm not
> >>> sure.
> >>
> >> Yes, that is correct. Raisin won't try to use sudo before asking the
> >> user first. That is the expected behaviour, if it doesn't work that way
> >> is a bug.
> >>
> >> Moreover I would be happy to introduce signature checks on git clones
> >> and downloads in Raisin.
> >>
> >
> > Does it have the mode Ian mentions where it will just print the depends
> > out to the user instead of installing them for you?
>
> Not exactly. At the moment it will determine, based on your OS and
> what you're trying to compile, which packages you need, and of those
> which packages are not installed, to generate a list of packages to
> install. If you pass "-v", and require a password for sudo, it will
> print the list it is about to install before prompting you for the
> password, at which point you could copy & paste if you want to.
>
> So all the functionality is there internally, it's just a matter of
> adding commands to make it only print the list rather than trying to
> actually install them.
There is no need to pass "-v" to get raisin to print the dependency
list. Also, unless the user explicitly says "yes", raisin won't invoke
sudo.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [qubes-devel] Re: Critique of the Xen Security Process
2015-11-10 2:46 ` [qubes-devel] " Radoslaw Szkodzinski
@ 2015-11-11 20:14 ` chris
0 siblings, 0 replies; 25+ messages in thread
From: chris @ 2015-11-11 20:14 UTC (permalink / raw
To: Radoslaw Szkodzinski
Cc: qubes-devel@googlegroups.com, xen-devel@lists.xen.org
[-- Attachment #1.1: Type: text/plain, Size: 4128 bytes --]
This is true but I can see her point of view as someone with a product that
is based on the open source core. There have been a lot of very drastic
changes that would cause someone with a product to have to do a significant
amount of re-engineering. Also how many different toolstacks have their
been?
I also see both sides but Joanna definitely has some very good points
chris
On Mon, Nov 9, 2015 at 9:46 PM, Radoslaw Szkodzinski <astralstorm@gmail.com>
wrote:
> As usual. Security, performance, convenience, price. Pick any mixture.
>
> As is usual for most software, developer convenience trumps most other
> considerations. I include ease of generating nice papers and jobs
> under developer convenience.
> Big players are much more concerned about performance, which saves
> money on machines you need to buy. (Note the resources poured into
> tmem, which is very iffy from security standpoint.)
> Even if there was a sudden security drive for Xen, it's a huge and
> probably fruitless task. Xen is not designed top down so you get no
> nice overview to check main assumptions. This is also why Xen is
> written in C and assembly, not in anything easier to use.
>
> Likewise security-oriented microkernels such as seL4 push drivers to
> "userspace" making it convenient for their developers, at the cost of
> performance and security, since they are now Somebody Else's Problem.
> Surprisingly, seL4 seems to be the only active and working
> non-research project that happens to have security as main aim, and
> they don't even implement many of features that are "required" for
> modern secure design, say, IOMMU...
> They want you to write your drivers as automated proofs instead.
> Meaning cheap developers will not be able to even lift a finger in
> writing those, since you need to learn Isabelle DSL and understand
> what is required in order to write a working driver with a good API,
> then also understand and model features of underlying hardware. This
> requires good mathematical background, as in actual CS, not just
> writing code.
>
> What seL4 folks are actually doing (have done, even) is writing a
> potent compiler between both C and theorem proving language, plus a
> set of theorems describing assumptions made on underlying hardware and
> security features. Very simple, just some 250k lines of theorems to
> prove correctness of 7k lines of C code.
>
> Good luck trying to port Qubes to their architecture. It's not
> impossible, but it is quite a task. So many theorems to wade through
> and understand, not to mention a whole new programming language.
> You'd get to add well defined isolation primitives to their theorem
> prover, then write a parallel proof to capDL kernel (ouch) or extend
> capDL with such isolation features and requisite APIs.
> Again, developer convenience suffers and have too high barrier of
> entry, and you won't get any code written or it will get too expensive
> in terms of time, skill and therefore money.
>
> As to other attempts...
> Python code? Forget about it, Python is thoroughly undefined behavior
> with no truly defined memory model. :) Even C++11 is better there.
>
> Rust? Rust mechanisms are undefined in terms of timing, which will
> result in unbounded execution time (hangs) when actual hardware is
> involved - there is no way to describe timing behavior of hardware in
> it in such a way that these properties are verified at higher layers.
> They attempt concurrency but the primitives are not bound to actual
> real time and hardware. There is no library for bare metal
> programming. Memory allocations are assumed to never fail - and many
> other things as well. (Does this sound like some other "never fails"
> software we're depending upon?)
>
> Trying to write a secure kernel in an unsafe language is akin to
> trying to bail a sinking ship with a spoon. It can theoretically be
> done, just not in practice. And actual low-level and safe languages
> are surprisingly rare.
>
> Best regards,
> R.
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>
[-- Attachment #1.2: Type: text/html, Size: 4907 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 25+ messages in thread
end of thread, other threads:[~2015-11-11 20:14 UTC | newest]
Thread overview: 25+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-11-06 17:22 Critique of the Xen Security Process Joanna Rutkowska
2015-11-06 19:41 ` James Bulpin
2015-11-06 22:42 ` Low Eel
2015-11-07 16:51 ` w.peter.howell
2015-11-09 12:11 ` Jan Beulich
2015-11-09 16:31 ` [qubes-devel] " Franz
2015-11-09 18:15 ` Wojtek Porczyk
2015-11-10 13:09 ` Lars Kurth
2015-11-10 14:10 ` Franz
2015-11-10 10:52 ` Lars Kurth
2015-11-11 11:36 ` Chris Laprise
2015-11-11 15:24 ` Lars Kurth
2015-11-09 21:48 ` Doug Goldstein
2015-11-11 9:43 ` Ian Campbell
2015-11-11 9:59 ` Lars Kurth
2015-11-11 17:21 ` Lars Kurth
2015-11-11 12:33 ` Raisin, was " Stefano Stabellini
2015-11-11 16:24 ` Doug Goldstein
2015-11-11 17:40 ` George Dunlap
2015-11-11 17:49 ` Stefano Stabellini
2015-11-11 12:34 ` Wei Liu
2015-11-09 22:00 ` chris
2015-11-10 2:46 ` [qubes-devel] " Radoslaw Szkodzinski
2015-11-11 20:14 ` chris
2015-11-11 12:59 ` Stefano Stabellini
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.