* [RFC PATCH] ARM: uprobes need icache flush after xol write
@ 2014-04-08 3:04 Victor Kamensky
2014-04-08 3:04 ` Victor Kamensky
0 siblings, 1 reply; 17+ messages in thread
From: Victor Kamensky @ 2014-04-08 3:04 UTC (permalink / raw
To: linux-arm-kernel
Hi Dave, Oleg, and All,
Short story
===========
It seems to me that ARMv7 uprobes need proper icache
flush after xol write. Please look at [1] discussion for similar
issue on ppc.
It seems that flush_dcache_page was sufficient for latter
architectures of PPC but it does not look that it is good enough
for ARMv7.
AFAIK know ARM V7 does not have "snooping Harvard caches"
and needs something like __cpuc_coherent_user_range function call
to sync up icache and dcache after instruction write through
dcache.
Patch that I propose follows this cover letter. There I
introduced weak arch_uprobe_xol_sync_dcache_icache function that
does traditional flush_dcache_page call and I redefined this
function to __cpuc_coherent_user_range call in ARM v7 > case.
[1] http://linux-kernel.2935.n7.nabble.com/Re-PATCH-6-9-uprobes-flush-cache-after-xol-write-td216886.html
Longer story
============
I was trying Dave's armv7 uprobes with SystemTap on Arndale
board. I used Linaro linux branch 3.14 based that contained
Dave's armv7 uprobes topic code. I believe it should be
pretty much the same as armv7 uprobes code that went to Russell's
tree.
I was able to do one function simple test - it worked
fine for me. But when I've tried to run many function like "probe
process("foobar").function("*")" probe SystemTap my target
process always crashed.
After quite a bit of chasing the issue, I was able to come
up with test case that shows several probes installed against
'ls' process. First probe placed at 'push {r4, r5, r6, r7,
r8, r9, r10, r11, lr}' instruction, which is first in
_getopt_initialize function, then script adds few more probes
at _getopt_initialize addresses that are executed latter. And
in those probes I dump registers set and top of stack. By
looking at execution of script one may easily conclude that it
looks like that for each probe 'push {r4, r5, r6, r7, r8, r9,
r10, r11, lr}' instruction is always executed - one may see
36 bytes increase of stack size and see copy of corresponding
registers on the stack.
The code path is the following:
handle_swbp -> pre_ssout
pre_ssout -> xol_get_insn_slot
xol_get_insn_slot -> copy_to_page
xol_get_insn_slot -> flush_dcache_page
pre_ssout -> arch_uprobe_pre_xol
pre_ssout function calls xol_get_insn_slot which finds
available slot in XOL area, that is mapped into user process
and copies required instruction into xol slot. After that it
calls flush_dcache_page, but icache is not flushed in ARM
case by this function. So I think the following thing happens:
first time first xol slot got 'push {r4, r5, r6, r7, r8, r9,
r10, r11, lr}' instruction and it retrieved into icache. Latter
when other probes are executed the same first slot of xol area
it will get different instruction but because icache is not
flushed CPU keep executing 'push' instruction.
When I add the following testing patch that flush icache
in arch_uprobe_pre_xol
[kamensky at kamensky-w530 git]$ git diff
diff --git a/arch/arm/kernel/uprobes.c b/arch/arm/kernel/uprobes.c
index f9bacee..ef34623 100644
--- a/arch/arm/kernel/uprobes.c
+++ b/arch/arm/kernel/uprobes.c
@@ -117,6 +117,8 @@ int arch_uprobe_pre_xol(struct arch_uprobe *auprobe, struct pt_regs *regs)
{
struct uprobe_task *utask = current->utask;
+ __flush_icache_all();
+
if (auprobe->prehandler)
auprobe->prehandler(auprobe, &utask->autask, regs);
it fixes my test problem, 'ls' executes fine, I am able to
trace it with "process("foobar").function("*")" traces.
Having complete icache flush is far from optimal. I was
looking for better and more optimal and targeted icache flush.
In discussion [1] and corresponding commit it was alluded
that flush_icache_user_range should help but I don't think it
will work for ARMv7. It seems that for ARMv7
flush_icache_user_range is not quite correct - it calls
again 'flush_dcache_page(page)' which I don't think touches
icache (other strange thing that it ignores len completely).
I looked at armv7 kprobes code, in similar area arch_prepare_kprobe
function calls arch_prepare_kprobe which calls flush_insn macro
which is defined as call flush_icache_range, which in turn
defines as __cpuc_coherent_kern_range(s,e).
I looked at ptrace breakpoint write code as well that deals
with similar issue.
As far as I see it seems that the best function to sync up icache
and dcache of user land process is to use __cpuc_coherent_user_range
function which sync up dcache and icache memory region of given
size and at given address on current core. In case of Arndale
it goes through v7_coherent_user_range function. Given uprobes
single step like use case it seem syncing up cache on current core
is sufficient. If someone can confirm that __cpuc_coherent_user_range
is right choice for this situation it would be nice. Test shows
it work, we would need to know for sure.
Next issue is how to integrate this call with cpu independent
uprobes code. I introduced week arch_uprobe_xol_sync_dcache_icache
function that calls flush_dcache_page as before, and for ARMv7
defined one that calls __cpuc_coherent_user_range. As far I
understand flush_dcache_page was introduced for some relatively
recent ppc CPU, it seems to be noop on x86. I was thinking that
default weak version of arch_uprobe_xol_sync_dcache_icache should
be empty and ppc define one that calls flush_dcache_page. But
I have no way to test ppc case so I decided do conservative
implementation and keep default version calling flush_dcache_page
as it was before.
Thanks,
Victor
Appendix: Test case that shows push instruction executed several times
======================================================================
SystemTap test script
---------------------
root at genericarmv7a:~/systemtap/test# cat ls_t4_not_r4.stp
function print_memory(addr:long, size:long) {
i = 0;
addr2 = addr;
while (i < size) {
printf("0x%8.8x: ", addr2);
while (i < size) {
printf ("0x%8.8x ", user_uint32(addr2));
addr2 = addr2 + 4;
i = i + 4;
if (i%16 == 0) {
break;
}
}
printf("\n");
}
}
probe process("/bin/ls.coreutils").function("_getopt_initialize")
{
sp = register("sp");
print_memory(sp, 64);
print_regs();
printf("-> _getopt_initialize\n");
}
probe process("/bin/ls.coreutils").statement(0x0001b2e8) {
sp = register("sp");
print_memory(sp, 64);
print_regs();
printf("-> 0x0001b2e8\n");
}
probe process("/bin/ls.coreutils").statement(0x0001b2d8) {
sp = register("sp");
print_memory(sp, 64);
print_regs();
printf("-> 0x0001b2d8\n");
}
probe process("/bin/ls.coreutils").statement(0x0001b408) {
sp = register("sp");
print_memory(sp, 64);
print_regs();
printf("-> 0x0001b408\n");
}
probe process("/bin/ls.coreutils").function("_getopt_internal_r")
{
// sp = register("sp");
// print_memory(sp, 64);
print_regs();
printf("-> _getopt_internal_r\n");
}
execution of script
-------------------
Look at log of script execution. Check how $sp changes at
each probe (+36). And see that r4, r5, r6, r7, r8, r9, r10, r11,
lr registers are always at top of stack.
root at genericarmv7a:~/systemtap/test# stap -U -v ls_t4_not_r4.stp
Pass 1: parsed user script and 100 library script(s) using 20520virt/16336res/1728shr/15260data kb, in 410usr/30sys/437real ms.
Pass 2: analyzed script: 5 probe(s), 8 function(s), 3 embed(s), 2 global(s) using 21984virt/18568res/2620shr/16724data kb, in 1280usr/870sys/2507real ms.
Pass 3: translated to C into "/tmp/stapPmXkE6/stap_06d9647b8c7b3327fecfc3259c39e0ed_6448_src.c" using 21984virt/18796res/2848shr/16724data kb, in 70usr/260sys/330real ms.
Pass 4: compiled C into "stap_06d9647b8c7b3327fecfc3259c39e0ed_6448.ko" in 16050usr/1290sys/18328real ms.
Pass 5: starting run.
CPU: 1pc : [<0001b2cc>] lr : [<0001c1ac>]
sp : 7eeb2940 ip : 00000001 fp : 0002a2d8
r10: 76ffe000 r9 : 0002a2d8 r8 : 7eeb29c4
r7 : 00000000 r6 : 00000000 r5 : 0002a2b0 r4 : 0002af40
r3 : 0001e914 r2 : 00020dc4 r1 : 7eeb2de4 r0 : 00000001
Flags: nZCv IRQs on FIQs on Mode USER_32 Segment user
Control: 30C5387D
Table: AC5A14C0 DAC: 55555555
-> _getopt_internal_r
0x7eeb28d8: 0xffffffff 0x00000000 0x76e97e34 0x76ffb8f8
0x7eeb28e8: 0x00000000 0x00000038 0x00000000 0x76f560c8
0x7eeb28f8: 0x00000077 0x00001500 0x00000005 0x000018b2
0x7eeb2908: 0x00000a3b 0x7f1c0300 0x01000415 0x76ffa4c0
CPU: 1pc : [<0001b2d8>] lr : [<0001c1ac>]
sp : 7eeb28d8 ip : 00000001 fp : 0002a2d8
r10: 00000001 r9 : 0002a2d8 r8 : 7eeb29c4
r7 : 00000000 r6 : 00000000 r5 : 0002a2b0 r4 : 0002af40
r3 : 0001e914 r2 : 00020dc4 r1 : 7eeb2de4 r0 : 00000001
Flags: nzCv IRQs on FIQs on Mode USER_32 Segment user
Control: 30C5387D
Table: AC5A14C0 DAC: 55555555
-> 0x0001b2d8
0x7eeb28b4: 0x0002af40 0x0002a2b0 0x00000000 0x00000000
0x7eeb28c4: 0x7eeb29c4 0x0002a2d8 0x7eeb2de4 0x0001e914
0x7eeb28d4: 0x0001c1ac 0xffffffff 0x00000000 0x76e97e34
0x7eeb28e4: 0x76ffb8f8 0x00000000 0x00000038 0x00000000
CPU: 1pc : [<0001b2e8>] lr : [<0001c1ac>]
sp : 7eeb28b4 ip : 00000001 fp : 0002a2d8
r10: 00000001 r9 : 0002a2d8 r8 : 7eeb29c4
r7 : 00000000 r6 : 00000000 r5 : 0002a2b0 r4 : 0002af40
r3 : 00000001 r2 : 00020dc4 r1 : 7eeb2de4 r0 : 00000001
Flags: nzCv IRQs on FIQs on Mode USER_32 Segment user
Control: 30C5387D
Table: AC5A14C0 DAC: 55555555
-> 0x0001b2e8
0x7eeb2890: 0x0002af40 0x0002a2b0 0x00000000 0x00000000
0x7eeb28a0: 0x7eeb29c4 0x00000001 0x00000001 0x0002a2d8
0x7eeb28b0: 0x0001c1ac 0x0002af40 0x0002a2b0 0x00000000
0x7eeb28c0: 0x00000000 0x7eeb29c4 0x0002a2d8 0x7eeb2de4
CPU: 1pc : [<0001b3f8>] lr : [<0001c1ac>]
sp : 7eeb2890 ip : 00000001 fp : 0002a2d8
r10: 00000001 r9 : 0002a2d8 r8 : 7eeb29c4
r7 : 00000000 r6 : 00000000 r5 : 0002a2b0 r4 : 0002af40
r3 : 00000001 r2 : 00000000 r1 : 7eeb2de4 r0 : 00000001
Flags: nZCv IRQs on FIQs on Mode USER_32 Segment user
Control: 30C5387D
Table: AC5A14C0 DAC: 55555555
-> _getopt_initialize
0x7eeb286c: 0x0002af40 0x0002a2b0 0x00000000 0x00000000
0x7eeb287c: 0x7eeb29c4 0x0002a2d8 0x00000001 0x0002a2d8
0x7eeb288c: 0x0001c1ac 0x0002af40 0x0002a2b0 0x00000000
0x7eeb289c: 0x00000000 0x7eeb29c4 0x00000001 0x00000001
CPU: 1pc : [<0001b408>] lr : [<0001c1ac>]
sp : 7eeb286c ip : 00000001 fp : 0002a2d8
r10: 00000001 r9 : 0002a2d8 r8 : 7eeb29c4
r7 : 00000000 r6 : 00000000 r5 : 0002a2b0 r4 : 0002af40
r3 : 00000001 r2 : 00000000 r1 : 7eeb2de4 r0 : 00000001
Flags: nzCv IRQs on FIQs on Mode USER_32 Segment user
Control: 30C5387D
Table: AC5A14C0 DAC: 55555555
-> 0x0001b408
gdb session of crashed ls command
=================================
Looking at core of crashed ls process. Look at
disassemble of function you can see uprobe breakpoints
as <UNDEFINED> instruction. For instructions that should
be there and which are executed through xol look at next
section.
root at genericarmv7a:~# gdb /bin/ls.coreutils -c core
GNU gdb (Linaro GDB) 7.6.1-2013.10
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-oe-linux-gnueabi".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from /bin/ls.coreutils...Reading symbols from /bin/.debug/ls.coreutils...done.
done.
warning: core file may not match specified executable file.
[New LWP 12424]
Core was generated by `ls'.
Program terminated with signal 11, Segmentation fault.
#0 _getopt_initialize (argc=1, argv=0x7eeb29c4, posixly_correct=175936, d=0x2af40 <getopt_data>,
optstring=0x2a2b0 <rpl_optind> "\001") at lib/getopt.c:241
241 if (optstring[0] == '-')
(gdb) bt
#0 _getopt_initialize (argc=1, argv=0x7eeb29c4, posixly_correct=175936, d=0x2af40 <getopt_data>,
optstring=0x2a2b0 <rpl_optind> "\001") at lib/getopt.c:241
#1 _getopt_internal_r (argc=1, argv=0x7eeb29c4, optstring=0x2a2b0 <rpl_optind> "\001",
longopts=0x2a2d8 <interrupt_signal>, longind=0x1c1ac <rpl_getopt_internal+72>,
long_only=175936, d=0x2a2b0 <rpl_optind>, posixly_correct=0) at lib/getopt.c:361
#2 0x0002a2d8 in ?? ()
Cannot access memory at address 0x1
#3 0x0002a2d8 in ?? ()
Cannot access memory at address 0x1
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) info reg
r0 0x1 1
r1 0x7eeb2de4 2129341924
r2 0x0 0
r3 0x1 1
r4 0x2af40 175936
r5 0x2a2b0 172720
r6 0x0 0
r7 0x0 0
r8 0x7eeb29c4 2129340868
r9 0x2a2d8 172760
r10 0x1 1
r11 0x2a2d8 172760
r12 0x0 0
sp 0x7eeb2848 0x7eeb2848
lr 0x1c1ac 115116
pc 0x1b420 0x1b420 <_getopt_internal_r+340>
cpsr 0x200f0010 537853968
(gdb) set height 0
(gdb) disassemble _getopt_internal_r
Dump of assembler code for function _getopt_internal_r:
0x0001b2cc <+0>: ; <UNDEFINED> instruction: 0xe7f001f9
0x0001b2d0 <+4>: sub sp, sp, #68 ; 0x44
0x0001b2d4 <+8>: subs r10, r0, #0
0x0001b2d8 <+12>: ; <UNDEFINED> instruction: 0xe7f001f9
0x0001b2dc <+16>: str r3, [sp, #28]
0x0001b2e0 <+20>: str r1, [sp, #24]
0x0001b2e4 <+24>: ldr r3, [r4, #4]
0x0001b2e8 <+28>: ; <UNDEFINED> instruction: 0xe7f001f9
0x0001b2ec <+32>: str r3, [sp, #20]
0x0001b2f0 <+36>: ble 0x1b4f8 <_getopt_internal_r+556>
0x0001b2f4 <+40>: ldr r3, [r4]
0x0001b2f8 <+44>: mov r2, #0
0x0001b2fc <+48>: str r2, [r4, #12]
0x0001b300 <+52>: cmp r3, r2
0x0001b304 <+56>: beq 0x1b3f0 <_getopt_internal_r+292>
0x0001b308 <+60>: ldr r2, [r4, #16]
0x0001b30c <+64>: cmp r2, #0
0x0001b310 <+68>: beq 0x1b3f8 <_getopt_internal_r+300>
0x0001b314 <+72>: ldr r5, [sp, #12]
0x0001b318 <+76>: ldrb r3, [r5]
0x0001b31c <+80>: cmp r3, #45 ; 0x2d
0x0001b320 <+84>: cmpne r3, #43 ; 0x2b
0x0001b324 <+88>: ldrbeq r3, [r5, #1]
0x0001b328 <+92>: addeq r5, r5, #1
0x0001b32c <+96>: streq r5, [sp, #12]
0x0001b330 <+100>: cmp r3, #58 ; 0x3a
0x0001b334 <+104>: ldr r9, [r4, #20]
0x0001b338 <+108>: ldr r12, [sp, #20]
0x0001b33c <+112>: moveq r12, #0
0x0001b340 <+116>: cmp r9, #0
0x0001b344 <+120>: str r12, [sp, #20]
0x0001b348 <+124>: beq 0x1b458 <_getopt_internal_r+396>
0x0001b34c <+128>: ldrb r3, [r9]
0x0001b350 <+132>: cmp r3, #0
0x0001b354 <+136>: beq 0x1b458 <_getopt_internal_r+396>
0x0001b358 <+140>: str r9, [sp, #16]
0x0001b35c <+144>: ldr r12, [sp, #28]
0x0001b360 <+148>: cmp r12, #0
0x0001b364 <+152>: beq 0x1b880 <_getopt_internal_r+1460>
0x0001b368 <+156>: ldr r3, [r4]
0x0001b36c <+160>: ldr r5, [sp, #24]
0x0001b370 <+164>: str r3, [sp, #32]
0x0001b374 <+168>: ldr r3, [r5, r3, lsl #2]
0x0001b378 <+172>: ldrb r1, [r3, #1]
0x0001b37c <+176>: cmp r1, #45 ; 0x2d
0x0001b380 <+180>: beq 0x1b564 <_getopt_internal_r+664>
0x0001b384 <+184>: ldr r12, [sp, #108] ; 0x6c
0x0001b388 <+188>: cmp r12, #0
0x0001b38c <+192>: bne 0x1b548 <_getopt_internal_r+636>
0x0001b390 <+196>: ldr r9, [sp, #16]
0x0001b394 <+200>: add r6, r9, #1
0x0001b398 <+204>: str r6, [r4, #20]
0x0001b39c <+208>: ldrb r5, [r9]
0x0001b3a0 <+212>: ldr r0, [sp, #12]
0x0001b3a4 <+216>: mov r1, r5
0x0001b3a8 <+220>: bl 0x9db4 <strchr>
0x0001b3ac <+224>: ldrb r3, [r6]
0x0001b3b0 <+228>: cmp r3, #0
0x0001b3b4 <+232>: ldreq r3, [r4]
0x0001b3b8 <+236>: addeq r3, r3, #1
0x0001b3bc <+240>: streq r3, [r4]
0x0001b3c0 <+244>: sub r3, r5, #58 ; 0x3a
0x0001b3c4 <+248>: cmp r0, #0
0x0001b3c8 <+252>: cmpne r3, #1
0x0001b3cc <+256>: bhi 0x1b6f0 <_getopt_internal_r+1060>
0x0001b3d0 <+260>: ldr r12, [sp, #20]
0x0001b3d4 <+264>: cmp r12, #0
0x0001b3d8 <+268>: bne 0x1b6b0 <_getopt_internal_r+996>
0x0001b3dc <+272>: mov r1, #63 ; 0x3f
0x0001b3e0 <+276>: str r5, [r4, #8]
0x0001b3e4 <+280>: mov r0, r1
0x0001b3e8 <+284>: add sp, sp, #68 ; 0x44
0x0001b3ec <+288>: pop {r4, r5, r6, r7, r8, r9, r10, r11, pc}
0x0001b3f0 <+292>: mov r3, #1
0x0001b3f4 <+296>: str r3, [r4]
0x0001b3f8 <+300>: ; <UNDEFINED> instruction: 0xe7f001f9
0x0001b3fc <+304>: str r3, [r4, #36] ; 0x24
0x0001b400 <+308>: cmp r5, #0
0x0001b404 <+312>: str r3, [r4, #32]
0x0001b408 <+316>: ; <UNDEFINED> instruction: 0xe7f001f9
0x0001b40c <+320>: str r3, [r4, #20]
0x0001b410 <+324>: movne r0, #1
0x0001b414 <+328>: beq 0x1b530 <_getopt_internal_r+612>
0x0001b418 <+332>: ldr r12, [sp, #12]
0x0001b41c <+336>: str r0, [r4, #28]
=> 0x0001b420 <+340>: ldrb r3, [r12]
0x0001b424 <+344>: cmp r3, #45 ; 0x2d
0x0001b428 <+348>: beq 0x1b848 <_getopt_internal_r+1404>
0x0001b42c <+352>: cmp r3, #43 ; 0x2b
0x0001b430 <+356>: beq 0x1b868 <_getopt_internal_r+1436>
0x0001b434 <+360>: cmp r0, #0
disassemble of _getopt_internal_r function
------------------------------------------
Just to see what instructions got breakpoint
(gdb) disassemble _getopt_internal_r
Dump of assembler code for function _getopt_internal_r:
0x0001b2cc <+0>: push {r4, r5, r6, r7, r8, r9, r10, r11, lr}
0x0001b2d0 <+4>: sub sp, sp, #68 ; 0x44
0x0001b2d4 <+8>: subs r10, r0, #0
0x0001b2d8 <+12>: ldr r4, [sp, #112] ; 0x70
0x0001b2dc <+16>: str r3, [sp, #28]
0x0001b2e0 <+20>: str r1, [sp, #24]
0x0001b2e4 <+24>: ldr r3, [r4, #4]
0x0001b2e8 <+28>: str r2, [sp, #12]
0x0001b2ec <+32>: str r3, [sp, #20]
0x0001b2f0 <+36>: ble 0x1b4f8 <_getopt_internal_r+556>
0x0001b2f4 <+40>: ldr r3, [r4]
0x0001b2f8 <+44>: mov r2, #0
0x0001b2fc <+48>: str r2, [r4, #12]
0x0001b300 <+52>: cmp r3, r2
0x0001b304 <+56>: beq 0x1b3f0 <_getopt_internal_r+292>
0x0001b308 <+60>: ldr r2, [r4, #16]
0x0001b30c <+64>: cmp r2, #0
0x0001b310 <+68>: beq 0x1b3f8 <_getopt_internal_r+300>
0x0001b314 <+72>: ldr r5, [sp, #12]
0x0001b318 <+76>: ldrb r3, [r5]
0x0001b31c <+80>: cmp r3, #45 ; 0x2d
0x0001b320 <+84>: cmpne r3, #43 ; 0x2b
0x0001b324 <+88>: ldrbeq r3, [r5, #1]
0x0001b328 <+92>: addeq r5, r5, #1
0x0001b32c <+96>: streq r5, [sp, #12]
0x0001b330 <+100>: cmp r3, #58 ; 0x3a
0x0001b334 <+104>: ldr r9, [r4, #20]
0x0001b338 <+108>: ldr r12, [sp, #20]
0x0001b33c <+112>: moveq r12, #0
0x0001b340 <+116>: cmp r9, #0
0x0001b344 <+120>: str r12, [sp, #20]
0x0001b348 <+124>: beq 0x1b458 <_getopt_internal_r+396>
0x0001b34c <+128>: ldrb r3, [r9]
0x0001b350 <+132>: cmp r3, #0
0x0001b354 <+136>: beq 0x1b458 <_getopt_internal_r+396>
0x0001b358 <+140>: str r9, [sp, #16]
0x0001b35c <+144>: ldr r12, [sp, #28]
0x0001b360 <+148>: cmp r12, #0
0x0001b364 <+152>: beq 0x1b880 <_getopt_internal_r+1460>
0x0001b368 <+156>: ldr r3, [r4]
0x0001b36c <+160>: ldr r5, [sp, #24]
0x0001b370 <+164>: str r3, [sp, #32]
0x0001b374 <+168>: ldr r3, [r5, r3, lsl #2]
0x0001b378 <+172>: ldrb r1, [r3, #1]
0x0001b37c <+176>: cmp r1, #45 ; 0x2d
0x0001b380 <+180>: beq 0x1b564 <_getopt_internal_r+664>
0x0001b384 <+184>: ldr r12, [sp, #108] ; 0x6c
0x0001b388 <+188>: cmp r12, #0
0x0001b38c <+192>: bne 0x1b548 <_getopt_internal_r+636>
0x0001b390 <+196>: ldr r9, [sp, #16]
0x0001b394 <+200>: add r6, r9, #1
0x0001b398 <+204>: str r6, [r4, #20]
0x0001b39c <+208>: ldrb r5, [r9]
0x0001b3a0 <+212>: ldr r0, [sp, #12]
0x0001b3a4 <+216>: mov r1, r5
0x0001b3a8 <+220>: bl 0x9db4 <strchr>
0x0001b3ac <+224>: ldrb r3, [r6]
0x0001b3b0 <+228>: cmp r3, #0
0x0001b3b4 <+232>: ldreq r3, [r4]
0x0001b3b8 <+236>: addeq r3, r3, #1
0x0001b3bc <+240>: streq r3, [r4]
0x0001b3c0 <+244>: sub r3, r5, #58 ; 0x3a
0x0001b3c4 <+248>: cmp r0, #0
0x0001b3c8 <+252>: cmpne r3, #1
0x0001b3cc <+256>: bhi 0x1b6f0 <_getopt_internal_r+1060>
0x0001b3d0 <+260>: ldr r12, [sp, #20]
0x0001b3d4 <+264>: cmp r12, #0
0x0001b3d8 <+268>: bne 0x1b6b0 <_getopt_internal_r+996>
0x0001b3dc <+272>: mov r1, #63 ; 0x3f
0x0001b3e0 <+276>: str r5, [r4, #8]
0x0001b3e4 <+280>: mov r0, r1
0x0001b3e8 <+284>: add sp, sp, #68 ; 0x44
0x0001b3ec <+288>: pop {r4, r5, r6, r7, r8, r9, r10, r11, pc}
0x0001b3f0 <+292>: mov r3, #1
0x0001b3f4 <+296>: str r3, [r4]
0x0001b3f8 <+300>: ldr r5, [sp, #116] ; 0x74
0x0001b3fc <+304>: str r3, [r4, #36] ; 0x24
0x0001b400 <+308>: cmp r5, #0
0x0001b404 <+312>: str r3, [r4, #32]
0x0001b408 <+316>: mov r3, #0
0x0001b40c <+320>: str r3, [r4, #20]
0x0001b410 <+324>: movne r0, #1
0x0001b414 <+328>: beq 0x1b530 <_getopt_internal_r+612>
0x0001b418 <+332>: ldr r12, [sp, #12]
0x0001b41c <+336>: str r0, [r4, #28]
0x0001b420 <+340>: ldrb r3, [r12]
0x0001b424 <+344>: cmp r3, #45 ; 0x2d
0x0001b428 <+348>: beq 0x1b848 <_getopt_internal_r+1404>
0x0001b42c <+352>: cmp r3, #43 ; 0x2b
0x0001b430 <+356>: beq 0x1b868 <_getopt_internal_r+1436>
0x0001b434 <+360>: cmp r0, #0
Victor Kamensky (1):
ARM: uprobes need icache flush after xol write
arch/arm/kernel/uprobes.c | 6 ++++++
include/linux/uprobes.h | 3 +++
kernel/events/uprobes.c | 20 +++++++++++++++-----
3 files changed, 24 insertions(+), 5 deletions(-)
--
1.8.1.4
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [RFC PATCH] ARM: uprobes need icache flush after xol write
2014-04-08 3:04 [RFC PATCH] ARM: uprobes need icache flush after xol write Victor Kamensky
@ 2014-04-08 3:04 ` Victor Kamensky
2014-04-08 8:24 ` Dave Martin
0 siblings, 1 reply; 17+ messages in thread
From: Victor Kamensky @ 2014-04-08 3:04 UTC (permalink / raw
To: linux-arm-kernel
After instruction write into xol area, on ARM V7
architecture code need to flush icache to sync up dcache
and icache. Having just 'flush_dcache_page(page)' call is
not enough - it is possible to have stale instruction
sitting in icache for given xol area slot address.
Introduce arch_uprobe_xol_sync_dcache_icache weak function
that by default calls 'flush_dcache_page(page)' and on
ARM define one that calls __cpuc_coherent_user_range.
Signed-off-by: Victor Kamensky <victor.kamensky@linaro.org>
---
arch/arm/kernel/uprobes.c | 6 ++++++
include/linux/uprobes.h | 3 +++
kernel/events/uprobes.c | 20 +++++++++++++++-----
3 files changed, 24 insertions(+), 5 deletions(-)
diff --git a/arch/arm/kernel/uprobes.c b/arch/arm/kernel/uprobes.c
index f9bacee..841750b 100644
--- a/arch/arm/kernel/uprobes.c
+++ b/arch/arm/kernel/uprobes.c
@@ -113,6 +113,12 @@ int arch_uprobe_analyze_insn(struct arch_uprobe *auprobe, struct mm_struct *mm,
return 0;
}
+void arch_uprobe_xol_sync_dcache_icache(struct page *page,
+ unsigned long addr, unsigned long size)
+{
+ __cpuc_coherent_user_range(addr, size);
+}
+
int arch_uprobe_pre_xol(struct arch_uprobe *auprobe, struct pt_regs *regs)
{
struct uprobe_task *utask = current->utask;
diff --git a/include/linux/uprobes.h b/include/linux/uprobes.h
index edff2b9..2fe3dfd 100644
--- a/include/linux/uprobes.h
+++ b/include/linux/uprobes.h
@@ -32,6 +32,7 @@ struct vm_area_struct;
struct mm_struct;
struct inode;
struct notifier_block;
+struct page;
#define UPROBE_HANDLER_REMOVE 1
#define UPROBE_HANDLER_MASK 1
@@ -127,6 +128,8 @@ extern int arch_uprobe_exception_notify(struct notifier_block *self, unsigned l
extern void arch_uprobe_abort_xol(struct arch_uprobe *aup, struct pt_regs *regs);
extern unsigned long arch_uretprobe_hijack_return_addr(unsigned long trampoline_vaddr, struct pt_regs *regs);
extern bool __weak arch_uprobe_ignore(struct arch_uprobe *aup, struct pt_regs *regs);
+extern void __weak arch_uprobe_xol_sync_dcache_icache(struct page *page,
+ unsigned long addr, unsigned long size);
#else /* !CONFIG_UPROBES */
struct uprobes_state {
};
diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c
index 04709b6..0027883 100644
--- a/kernel/events/uprobes.c
+++ b/kernel/events/uprobes.c
@@ -1299,11 +1299,9 @@ static unsigned long xol_get_insn_slot(struct uprobe *uprobe)
/* Initialize the slot */
copy_to_page(area->page, xol_vaddr,
&uprobe->arch.ixol, sizeof(uprobe->arch.ixol));
- /*
- * We probably need flush_icache_user_range() but it needs vma.
- * This should work on supported architectures too.
- */
- flush_dcache_page(area->page);
+
+ arch_uprobe_xol_sync_dcache_icache(area->page,
+ xol_vaddr, sizeof(uprobe->arch.ixol));
return xol_vaddr;
}
@@ -1346,6 +1344,18 @@ static void xol_free_insn_slot(struct task_struct *tsk)
}
}
+void __weak arch_uprobe_xol_sync_dcache_icache(struct page *page,
+ unsigned long addr, unsigned long len)
+{
+ /*
+ * We probably need flush_icache_user_range() but it needs vma.
+ * This should work on most of architectures by default. If
+ * architecture needs to do something different it can define
+ * its own version of the function.
+ */
+ flush_dcache_page(page);
+}
+
/**
* uprobe_get_swbp_addr - compute address of swbp given post-swbp regs
* @regs: Reflects the saved state of the task after it has hit a breakpoint
--
1.8.1.4
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [RFC PATCH] ARM: uprobes need icache flush after xol write
2014-04-08 3:04 ` Victor Kamensky
@ 2014-04-08 8:24 ` Dave Martin
2014-04-08 11:46 ` Russell King - ARM Linux
0 siblings, 1 reply; 17+ messages in thread
From: Dave Martin @ 2014-04-08 8:24 UTC (permalink / raw
To: linux-arm-kernel
On Mon, Apr 07, 2014 at 08:04:20PM -0700, Victor Kamensky wrote:
> After instruction write into xol area, on ARM V7
> architecture code need to flush icache to sync up dcache
> and icache. Having just 'flush_dcache_page(page)' call is
> not enough - it is possible to have stale instruction
> sitting in icache for given xol area slot address.
>
> Introduce arch_uprobe_xol_sync_dcache_icache weak function
> that by default calls 'flush_dcache_page(page)' and on
> ARM define one that calls __cpuc_coherent_user_range.
You're right that something is needed.
Would flush_icache_range() not be the correct thing to call here?
Sync'ing the I and D sides for a whole page seems excessive to
install a probe.
I believe __cpuc_coherent_kern_range() is not intended as an API,
and should only be used in special situations in arch-specific code.
It's the correct operation, but arm provides flush_icache_range as a
wrapper to call.
flush_icache_range() is documented, with the correct effect, in
Documentation/cachetlb.txt, so I believe that all arches should
provide it, even if it's a no-op.
Cheers
---Dave
>
> Signed-off-by: Victor Kamensky <victor.kamensky@linaro.org>
> ---
> arch/arm/kernel/uprobes.c | 6 ++++++
> include/linux/uprobes.h | 3 +++
> kernel/events/uprobes.c | 20 +++++++++++++++-----
> 3 files changed, 24 insertions(+), 5 deletions(-)
>
> diff --git a/arch/arm/kernel/uprobes.c b/arch/arm/kernel/uprobes.c
> index f9bacee..841750b 100644
> --- a/arch/arm/kernel/uprobes.c
> +++ b/arch/arm/kernel/uprobes.c
> @@ -113,6 +113,12 @@ int arch_uprobe_analyze_insn(struct arch_uprobe *auprobe, struct mm_struct *mm,
> return 0;
> }
>
> +void arch_uprobe_xol_sync_dcache_icache(struct page *page,
> + unsigned long addr, unsigned long size)
> +{
> + __cpuc_coherent_user_range(addr, size);
> +}
> +
> int arch_uprobe_pre_xol(struct arch_uprobe *auprobe, struct pt_regs *regs)
> {
> struct uprobe_task *utask = current->utask;
> diff --git a/include/linux/uprobes.h b/include/linux/uprobes.h
> index edff2b9..2fe3dfd 100644
> --- a/include/linux/uprobes.h
> +++ b/include/linux/uprobes.h
> @@ -32,6 +32,7 @@ struct vm_area_struct;
> struct mm_struct;
> struct inode;
> struct notifier_block;
> +struct page;
>
> #define UPROBE_HANDLER_REMOVE 1
> #define UPROBE_HANDLER_MASK 1
> @@ -127,6 +128,8 @@ extern int arch_uprobe_exception_notify(struct notifier_block *self, unsigned l
> extern void arch_uprobe_abort_xol(struct arch_uprobe *aup, struct pt_regs *regs);
> extern unsigned long arch_uretprobe_hijack_return_addr(unsigned long trampoline_vaddr, struct pt_regs *regs);
> extern bool __weak arch_uprobe_ignore(struct arch_uprobe *aup, struct pt_regs *regs);
> +extern void __weak arch_uprobe_xol_sync_dcache_icache(struct page *page,
> + unsigned long addr, unsigned long size);
> #else /* !CONFIG_UPROBES */
> struct uprobes_state {
> };
> diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c
> index 04709b6..0027883 100644
> --- a/kernel/events/uprobes.c
> +++ b/kernel/events/uprobes.c
> @@ -1299,11 +1299,9 @@ static unsigned long xol_get_insn_slot(struct uprobe *uprobe)
> /* Initialize the slot */
> copy_to_page(area->page, xol_vaddr,
> &uprobe->arch.ixol, sizeof(uprobe->arch.ixol));
> - /*
> - * We probably need flush_icache_user_range() but it needs vma.
> - * This should work on supported architectures too.
> - */
> - flush_dcache_page(area->page);
> +
> + arch_uprobe_xol_sync_dcache_icache(area->page,
> + xol_vaddr, sizeof(uprobe->arch.ixol));
>
> return xol_vaddr;
> }
> @@ -1346,6 +1344,18 @@ static void xol_free_insn_slot(struct task_struct *tsk)
> }
> }
>
> +void __weak arch_uprobe_xol_sync_dcache_icache(struct page *page,
> + unsigned long addr, unsigned long len)
> +{
> + /*
> + * We probably need flush_icache_user_range() but it needs vma.
> + * This should work on most of architectures by default. If
> + * architecture needs to do something different it can define
> + * its own version of the function.
> + */
> + flush_dcache_page(page);
> +}
> +
> /**
> * uprobe_get_swbp_addr - compute address of swbp given post-swbp regs
> * @regs: Reflects the saved state of the task after it has hit a breakpoint
> --
> 1.8.1.4
>
>
> _______________________________________________
> linaro-kernel mailing list
> linaro-kernel at lists.linaro.org
> http://lists.linaro.org/mailman/listinfo/linaro-kernel
^ permalink raw reply [flat|nested] 17+ messages in thread
* [RFC PATCH] ARM: uprobes need icache flush after xol write
2014-04-08 8:24 ` Dave Martin
@ 2014-04-08 11:46 ` Russell King - ARM Linux
2014-04-08 13:05 ` David Long
0 siblings, 1 reply; 17+ messages in thread
From: Russell King - ARM Linux @ 2014-04-08 11:46 UTC (permalink / raw
To: linux-arm-kernel
On Tue, Apr 08, 2014 at 09:24:50AM +0100, Dave Martin wrote:
> You're right that something is needed.
>
> Would flush_icache_range() not be the correct thing to call here?
>
> Sync'ing the I and D sides for a whole page seems excessive to
> install a probe.
>
> I believe __cpuc_coherent_kern_range() is not intended as an API,
> and should only be used in special situations in arch-specific code.
> It's the correct operation, but arm provides flush_icache_range as a
> wrapper to call.
>
> flush_icache_range() is documented, with the correct effect, in
> Documentation/cachetlb.txt, so I believe that all arches should
> provide it, even if it's a no-op.
The real question is, what is this code in xol_get_insn_slot() trying
to do?
1. area->page is an anonymous page - it was allocated by
alloc_page(GFP_HIGHUSER). This makes use of flush_dcache_page()
questionable at best.
2. xol_get_insn_slot() appears to want to copy instructions into this
page via the kernel mapping such that they're visible to userspace.
So, the first thing that we need to do is to push the written data out to
a point where the instruction stream can see it. Then we need to ensure
that the instruction stream can see the new instructions.
Practically for ARM, that means pushing it out of the D-cache and then
making sure that there are no I-cache lines associated with the new
instructions.
With VIVT instruction caches (yes, we still have them on ARMv7), we have
to flush the I-cache lines associated with the userspace addresses -
flushing the I-cache lines for the kernel mapping doesn't do anything for
the stale lines in the userspace mapping.
copy_to_page() kmap_atomic()'s the page - this doesn't take account of
VIPT aliasing caches, and so the writes may be in an alias of the user
page. Plus, when the page is kunmap_atomic()'d, there's no guarantee that
the written lines will be flushed out.
copy_from_page() suffers from a similar problem, though its saving grace
is that the user mapping is execute-only (iow, read+execute only) so
should never end up with any dirty cache lines associated with it.
flush_dcache_page() will ensure that the D-cache lines are flushed out on
ARM (even though it's only defined to have an effect for page cache pages)
but it does nothing for the I-cache. So, although it seems to partially
work, it's definitely the wrong interface here.
However... the normal way we do this (for example, in the case of ptrace)
is via copy_to_user_page() / copy_from_user_page(), and we have some
pretty complex handling via those functions to deal with writing to the
page and having the affected thread running on a different CPU to the
one we wrote the new instructions. I'd recommend that uprobes looks at
trying to re-use some of this infrastructure rather than re-inventing
this wheel.
--
FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
improving, and getting towards what was expected from it.
^ permalink raw reply [flat|nested] 17+ messages in thread
* [RFC PATCH] ARM: uprobes need icache flush after xol write
2014-04-08 11:46 ` Russell King - ARM Linux
@ 2014-04-08 13:05 ` David Long
2014-04-08 13:30 ` Russell King - ARM Linux
2014-04-08 14:15 ` Victor Kamensky
0 siblings, 2 replies; 17+ messages in thread
From: David Long @ 2014-04-08 13:05 UTC (permalink / raw
To: linux-arm-kernel
On 04/08/14 07:46, Russell King - ARM Linux wrote:
> On Tue, Apr 08, 2014 at 09:24:50AM +0100, Dave Martin wrote:
>> You're right that something is needed.
>>
>> Would flush_icache_range() not be the correct thing to call here?
>>
>> Sync'ing the I and D sides for a whole page seems excessive to
>> install a probe.
>>
>> I believe __cpuc_coherent_kern_range() is not intended as an API,
>> and should only be used in special situations in arch-specific code.
>> It's the correct operation, but arm provides flush_icache_range as a
>> wrapper to call.
>>
>> flush_icache_range() is documented, with the correct effect, in
>> Documentation/cachetlb.txt, so I believe that all arches should
>> provide it, even if it's a no-op.
>
> The real question is, what is this code in xol_get_insn_slot() trying
> to do?
>
> 1. area->page is an anonymous page - it was allocated by
> alloc_page(GFP_HIGHUSER). This makes use of flush_dcache_page()
> questionable at best.
>
> 2. xol_get_insn_slot() appears to want to copy instructions into this
> page via the kernel mapping such that they're visible to userspace.
>
> So, the first thing that we need to do is to push the written data out to
> a point where the instruction stream can see it. Then we need to ensure
> that the instruction stream can see the new instructions.
>
> Practically for ARM, that means pushing it out of the D-cache and then
> making sure that there are no I-cache lines associated with the new
> instructions.
>
> With VIVT instruction caches (yes, we still have them on ARMv7), we have
> to flush the I-cache lines associated with the userspace addresses -
> flushing the I-cache lines for the kernel mapping doesn't do anything for
> the stale lines in the userspace mapping.
>
> copy_to_page() kmap_atomic()'s the page - this doesn't take account of
> VIPT aliasing caches, and so the writes may be in an alias of the user
> page. Plus, when the page is kunmap_atomic()'d, there's no guarantee that
> the written lines will be flushed out.
>
> copy_from_page() suffers from a similar problem, though its saving grace
> is that the user mapping is execute-only (iow, read+execute only) so
> should never end up with any dirty cache lines associated with it.
>
> flush_dcache_page() will ensure that the D-cache lines are flushed out on
> ARM (even though it's only defined to have an effect for page cache pages)
> but it does nothing for the I-cache. So, although it seems to partially
> work, it's definitely the wrong interface here.
>
> However... the normal way we do this (for example, in the case of ptrace)
> is via copy_to_user_page() / copy_from_user_page(), and we have some
> pretty complex handling via those functions to deal with writing to the
> page and having the affected thread running on a different CPU to the
> one we wrote the new instructions. I'd recommend that uprobes looks at
> trying to re-use some of this infrastructure rather than re-inventing
> this wheel.
>
Conincentally I ran into this same problem last weekend while testing my
future thumb extensions to the uprobes code. I too tried a couple
cache-flushing changes, and found most any change fixed the problem and
the past intermittent systemtap failures I had seen (although I was
unsure how much of that was due to happy side-effects from any
particular flush operation). I found flush_icache_user_range() but I
was scratching my head on how to use it instead of a heavy-weight flush
as the uprobes code deliberately does not hang onto a copy of the user
vma pointer. I tried a patch storing the mm struct pointer in the
uprobes xol struct and looking up the vma info using it and the user
virtual address, but I'm not sure that's any better than just hanging
onto the vma pointer.
I also would not necessarily have assumed the kernel dcache flush was
still needed, but I now see how this makes sense. Thanks for the
explanation Russell, it answered several questions I had.
Unfortunately copy_to_user_page() also needs a pointer to a vma struct
so, while it presumably provides the model to follow, it can't simply be
dropped in.
Victor, do you want to own the action of coming up with a patch? If not
I'm OK with taking this forward, although I suspect both of us will need
help testing for anything other than ARM.
-dl
^ permalink raw reply [flat|nested] 17+ messages in thread
* [RFC PATCH] ARM: uprobes need icache flush after xol write
2014-04-08 13:05 ` David Long
@ 2014-04-08 13:30 ` Russell King - ARM Linux
2014-04-08 14:09 ` Victor Kamensky
2014-04-08 15:27 ` Oleg Nesterov
2014-04-08 14:15 ` Victor Kamensky
1 sibling, 2 replies; 17+ messages in thread
From: Russell King - ARM Linux @ 2014-04-08 13:30 UTC (permalink / raw
To: linux-arm-kernel
On Tue, Apr 08, 2014 at 09:05:49AM -0400, David Long wrote:
> Unfortunately copy_to_user_page() also needs a pointer to a vma struct
> so, while it presumably provides the model to follow, it can't simply be
> dropped in.
Well, isn't this code doing the same thing as ptrace? It seems to want
to modify a page in userspace of another process to change instructions
that are going to be executed. That's what ptrace does, and ptrace
already copes with all the issues there.
Given that we've already solved that problem, wouldn't it be a good idea
if the tracing code would stop trying to reinvent broken solutions to
problems we have already solved?
--
FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
improving, and getting towards what was expected from it.
^ permalink raw reply [flat|nested] 17+ messages in thread
* [RFC PATCH] ARM: uprobes need icache flush after xol write
2014-04-08 13:30 ` Russell King - ARM Linux
@ 2014-04-08 14:09 ` Victor Kamensky
2014-04-08 15:35 ` Victor Kamensky
2014-04-08 15:27 ` Oleg Nesterov
1 sibling, 1 reply; 17+ messages in thread
From: Victor Kamensky @ 2014-04-08 14:09 UTC (permalink / raw
To: linux-arm-kernel
On 8 April 2014 06:30, Russell King - ARM Linux <linux@arm.linux.org.uk> wrote:
> On Tue, Apr 08, 2014 at 09:05:49AM -0400, David Long wrote:
>> Unfortunately copy_to_user_page() also needs a pointer to a vma struct
>> so, while it presumably provides the model to follow, it can't simply be
>> dropped in.
>
> Well, isn't this code doing the same thing as ptrace? It seems to want
> to modify a page in userspace of another process to change instructions
> that are going to be executed. That's what ptrace does, and ptrace
> already copes with all the issues there.
As I see it, the difference between ptrace use and xol single stepping
it that in first all cores should be involved and potentially cache operation
would be broadcasted. In case of uprobes only local core is involved.
The way I read uprobes code xol slot will be used once by current core
to execute one instruction, then it will hit next breakpoint and xol slot
will be freed and potentially reused by next uprobe.
I assume that while doing xol single stepping task cannot migrate to
another core, wondering whether it is absolutely true. It seems that
similar ppc code that calls flush_dcache_page follow this assumption.
If uprobes xol single stepping need to handle all cores it will be very
expensive.
In ptrace case caches of all cores must be handled.
Having said above I do agree that large portion of flush_ptrace_access
could be reused between these two use cases.
Thanks,
Victor
> Given that we've already solved that problem, wouldn't it be a good idea
> if the tracing code would stop trying to reinvent broken solutions to
> problems we have already solved?
>
> --
> FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
> improving, and getting towards what was expected from it.
^ permalink raw reply [flat|nested] 17+ messages in thread
* [RFC PATCH] ARM: uprobes need icache flush after xol write
2014-04-08 13:05 ` David Long
2014-04-08 13:30 ` Russell King - ARM Linux
@ 2014-04-08 14:15 ` Victor Kamensky
1 sibling, 0 replies; 17+ messages in thread
From: Victor Kamensky @ 2014-04-08 14:15 UTC (permalink / raw
To: linux-arm-kernel
Hi David,
On 8 April 2014 06:05, David Long <dave.long@linaro.org> wrote:
>
> Victor, do you want to own the action of coming up with a patch? If not I'm
> OK with taking this forward, although I suspect both of us will need help
> testing for anything other than ARM.
Please take it forward. I can do some testing of patches you
will come up with.
Thanks,
Victor
^ permalink raw reply [flat|nested] 17+ messages in thread
* [RFC PATCH] ARM: uprobes need icache flush after xol write
2014-04-08 13:30 ` Russell King - ARM Linux
2014-04-08 14:09 ` Victor Kamensky
@ 2014-04-08 15:27 ` Oleg Nesterov
2014-04-08 15:41 ` Russell King - ARM Linux
1 sibling, 1 reply; 17+ messages in thread
From: Oleg Nesterov @ 2014-04-08 15:27 UTC (permalink / raw
To: linux-arm-kernel
Sorry, I din't read the whole thread yet...
On 04/08, Russell King - ARM Linux wrote:
> On Tue, Apr 08, 2014 at 09:05:49AM -0400, David Long wrote:
> > Unfortunately copy_to_user_page() also needs a pointer to a vma struct
> > so, while it presumably provides the model to follow, it can't simply be
> > dropped in.
>
> Well, isn't this code doing the same thing as ptrace? It seems to want
> to modify a page in userspace of another process to change instructions
> that are going to be executed. That's what ptrace does, and ptrace
> already copes with all the issues there.
Yes, but it does get_user_pages(&vma) and thus it knows vma.
> Given that we've already solved that problem, wouldn't it be a good idea
> if the tracing code would stop trying to reinvent broken solutions to
> problems we have already solved?
But uprobes can't do this. Of course, I am not saying this is impossible,
but it would be nice to avoid mmap_sem/find_vma/etc.
Almost nobody (iirc only sparc?) actually uses this "vma" arguments. And
at least the supported architectures do not (at least this is what I think
after the quick grep).
Perhaps we can rolerate the hack below?
Oleg.
--- x/kernel/events/uprobes.c
+++ x/kernel/events/uprobes.c
@@ -1295,10 +1295,10 @@ static unsigned long xol_get_insn_slot(s
copy_to_page(area->page, xol_vaddr,
&uprobe->arch.ixol, sizeof(uprobe->arch.ixol));
/*
- * We probably need flush_icache_user_range() but it needs vma.
- * This should work on supported architectures too.
+ * The architectures we currently support do not really use vma.
*/
- flush_dcache_page(area->page);
+ flush_icache_user_range(NULL /* vma */, area->page,
+ xol_vaddr, sizeof(uprobe->arch.ixol));
return xol_vaddr;
}
^ permalink raw reply [flat|nested] 17+ messages in thread
* [RFC PATCH] ARM: uprobes need icache flush after xol write
2014-04-08 14:09 ` Victor Kamensky
@ 2014-04-08 15:35 ` Victor Kamensky
2014-04-08 16:19 ` Russell King - ARM Linux
0 siblings, 1 reply; 17+ messages in thread
From: Victor Kamensky @ 2014-04-08 15:35 UTC (permalink / raw
To: linux-arm-kernel
On 8 April 2014 07:09, Victor Kamensky <victor.kamensky@linaro.org> wrote:
> On 8 April 2014 06:30, Russell King - ARM Linux <linux@arm.linux.org.uk> wrote:
>> On Tue, Apr 08, 2014 at 09:05:49AM -0400, David Long wrote:
>>> Unfortunately copy_to_user_page() also needs a pointer to a vma struct
>>> so, while it presumably provides the model to follow, it can't simply be
>>> dropped in.
>>
>> Well, isn't this code doing the same thing as ptrace? It seems to want
>> to modify a page in userspace of another process to change instructions
>> that are going to be executed. That's what ptrace does, and ptrace
>> already copes with all the issues there.
>
> As I see it, the difference between ptrace use and xol single stepping
> it that in first all cores should be involved and potentially cache operation
> would be broadcasted. In case of uprobes only local core is involved.
> The way I read uprobes code xol slot will be used once by current core
> to execute one instruction, then it will hit next breakpoint and xol slot
> will be freed and potentially reused by next uprobe.
>
> I assume that while doing xol single stepping task cannot migrate to
> another core, wondering whether it is absolutely true. It seems that
> similar ppc code that calls flush_dcache_page follow this assumption.
> If uprobes xol single stepping need to handle all cores it will be very
> expensive.
>
> In ptrace case caches of all cores must be handled.
>
> Having said above I do agree that large portion of flush_ptrace_access
> could be reused between these two use cases.
Looking at flush_ptrace_access more closely. Now I am not sure that
ptrace write code could easily reused.
1) flush_ptrace_access seems to handle both data and text segments
write. In case of xol write we always know that it is code write
2) as I pointed before flush_ptrace_access handles smp case whereas
xol write does not need to do that
3) flush_ptrace_access needs vma, which uprobe code does not have on
layer where xol is installed. That is probably most critical issue that
could stop suggested code sharing. And note that vma in
flush_ptrace_access is needed only for cases cases 1) and 2) above,
about which xol write does not really care. If we take only required
pieces from flush_ptrace_access would not xol cache flush look like
this:
void arch_uprobe_xol_sync_dcache_icache(struct page *page,
unsigned long addr, unsigned long size)
{
if (icache_is_vipt_aliasing())
flush_icache_alias(page_to_pfn(page), addr, size);
else
__cpuc_coherent_user_range(addr, size);
}
of course, above function should be properly places or flush_icache_alias
function should be made globally visible.
Note on the side: flush_ptrace_access works with user-land memory
but it still uses __cpuc_coherent_kern_range. I think it should use
__cpuc_coherent_user_range. It looks like implementation wise it is
the same but user variant seems more appropriate in given situation.
Or am I missing something here?
Thanks,
Victor
> Thanks,
> Victor
>
>> Given that we've already solved that problem, wouldn't it be a good idea
>> if the tracing code would stop trying to reinvent broken solutions to
>> problems we have already solved?
>>
>> --
>> FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
>> improving, and getting towards what was expected from it.
^ permalink raw reply [flat|nested] 17+ messages in thread
* [RFC PATCH] ARM: uprobes need icache flush after xol write
2014-04-08 15:27 ` Oleg Nesterov
@ 2014-04-08 15:41 ` Russell King - ARM Linux
2014-04-09 16:18 ` Russell King - ARM Linux
0 siblings, 1 reply; 17+ messages in thread
From: Russell King - ARM Linux @ 2014-04-08 15:41 UTC (permalink / raw
To: linux-arm-kernel
On Tue, Apr 08, 2014 at 05:27:35PM +0200, Oleg Nesterov wrote:
> Sorry, I din't read the whole thread yet...
>
> On 04/08, Russell King - ARM Linux wrote:
> > On Tue, Apr 08, 2014 at 09:05:49AM -0400, David Long wrote:
> > > Unfortunately copy_to_user_page() also needs a pointer to a vma struct
> > > so, while it presumably provides the model to follow, it can't simply be
> > > dropped in.
> >
> > Well, isn't this code doing the same thing as ptrace? It seems to want
> > to modify a page in userspace of another process to change instructions
> > that are going to be executed. That's what ptrace does, and ptrace
> > already copes with all the issues there.
>
> Yes, but it does get_user_pages(&vma) and thus it knows vma.
>
> > Given that we've already solved that problem, wouldn't it be a good idea
> > if the tracing code would stop trying to reinvent broken solutions to
> > problems we have already solved?
>
> But uprobes can't do this. Of course, I am not saying this is impossible,
> but it would be nice to avoid mmap_sem/find_vma/etc.
>
> Almost nobody (iirc only sparc?) actually uses this "vma" arguments. And
> at least the supported architectures do not (at least this is what I think
> after the quick grep).
>
> Perhaps we can rolerate the hack below?
This has no effect at fixing the reported problem though:
#define flush_icache_user_range(vma,page,addr,len) \
flush_dcache_page(page)
so it results in no change.
The bigger question is... what is this function supposed to do? It's
not been documented in Documentation/cachetlb.txt, and nothing in the
kernel refers to this function - it is completely unused.
I think let's start out by killing this function - the semantics of
this function have been lost, so it's not clear what it was supposed
to do in its original form. Even going back to the start of git
history, it looks like it was never used outside arch code.
--
FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
improving, and getting towards what was expected from it.
^ permalink raw reply [flat|nested] 17+ messages in thread
* [RFC PATCH] ARM: uprobes need icache flush after xol write
2014-04-08 15:35 ` Victor Kamensky
@ 2014-04-08 16:19 ` Russell King - ARM Linux
2014-04-08 16:29 ` David Long
2014-04-08 18:39 ` Victor Kamensky
0 siblings, 2 replies; 17+ messages in thread
From: Russell King - ARM Linux @ 2014-04-08 16:19 UTC (permalink / raw
To: linux-arm-kernel
On Tue, Apr 08, 2014 at 08:35:01AM -0700, Victor Kamensky wrote:
> Looking at flush_ptrace_access more closely. Now I am not sure that
> ptrace write code could easily reused.
>
> 1) flush_ptrace_access seems to handle both data and text segments
> write. In case of xol write we always know that it is code write
Of course it has to, but writing code is the harder of the two
problems. With writes to data segments, the only thing that has to
be dealt with is the data cache. With code, not only do you need to
deal with the data cache, but you also need to deal with the instruction
cache too.
> 2) as I pointed before flush_ptrace_access handles smp case whereas
> xol write does not need to do that
Are you sure about that?
If I'm reading the code correctly, uprobes inserts a trapping instruction
into the userspace program. When that instruction is hit, it checks
whether the thread is the desired one, and may request a slot in this
magic page, which is when the write happens.
The uprobes special page is shared across all threads which share the
mm_struct, so in the case of a multi-threaded program running on a SMP
machine, this page is visible to multiple CPUs.
Is it possible for uprobes to be active on more than one thread at a
time? If so, because that page is shared, you could end up writing
to a partial cache line from two threads. From what I can see, ixol[]
is two words, and there's normally 8 works per cache line on ARM, or
occasionally 16.
So, the question now is: is it possible to have uprobes active on more
than one thread, and for two threads to hit the uprobes processing, both
needing a slot in the page, hitting the same cache line?
Now, what happens if thread 1 on CPU1 gets there first with its write.
Then thread 2 on CPU2 gets there, causing the cache line to migrate to
CPU2. Then CPU1 does it's (non-broadcasted) flush, meanwhile CPU2 then
gets preempted and goes off and does something else.
Please tell me that can't happen. :)
> 3) flush_ptrace_access needs vma, which uprobe code does not have on
> layer where xol is installed. That is probably most critical issue that
> could stop suggested code sharing. And note that vma in
> flush_ptrace_access is needed only for cases cases 1) and 2) above,
> about which xol write does not really care. If we take only required
> pieces from flush_ptrace_access would not xol cache flush look like
> this:
>
> void arch_uprobe_xol_sync_dcache_icache(struct page *page,
> unsigned long addr, unsigned long size)
> {
> if (icache_is_vipt_aliasing())
> flush_icache_alias(page_to_pfn(page), addr, size);
> else
> __cpuc_coherent_user_range(addr, size);
> }
And this is only half the story.
What you're saying above is:
- if the *instruction* cache is an aliasing VIPT cache, then we can
map the user page at the same colour and flush the data and instruction
caches according to that colour.
Problem: if you have a VIPT aliasing data cache, and you wrote
to the page via a mapping which had a different colour
(which is highly likely given you're using kmap_atomic())
then you are not flushing the new instructions out of
the data cache.
- if the *instruction* cache is not an aliasing VIPT cache, then we
flush using the userspace mapping directly.
Problem: what if the data cache is aliasing... same problem
as the above case.
This is why we have the check for cache_is_vipt_aliasing() in there -
that's not _just_ to deal with ptrace data modification, it's there for
text modification as well.
> Note on the side: flush_ptrace_access works with user-land memory
> but it still uses __cpuc_coherent_kern_range. I think it should use
> __cpuc_coherent_user_range.
No, because __cpuc_coherent_kern_range() there is used on the *kernel*
mapping of the page, not on the *user* mapping (which may fault). The
kernel mapping will never fault.
> It looks like implementation wise it is
> the same but user variant seems more appropriate in given situation.
> Or am I missing something here?
They do seem to be the same for both cases, so we could probably remove
the distinction. It's unlikely we'll ever see an implementation needing
a difference between the two now.
--
FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
improving, and getting towards what was expected from it.
^ permalink raw reply [flat|nested] 17+ messages in thread
* [RFC PATCH] ARM: uprobes need icache flush after xol write
2014-04-08 16:19 ` Russell King - ARM Linux
@ 2014-04-08 16:29 ` David Long
2014-04-08 18:39 ` Victor Kamensky
1 sibling, 0 replies; 17+ messages in thread
From: David Long @ 2014-04-08 16:29 UTC (permalink / raw
To: linux-arm-kernel
On 04/08/14 12:19, Russell King - ARM Linux wrote:
> On Tue, Apr 08, 2014 at 08:35:01AM -0700, Victor Kamensky wrote:
>> Looking at flush_ptrace_access more closely. Now I am not sure that
>> ptrace write code could easily reused.
>>
>> 1) flush_ptrace_access seems to handle both data and text segments
>> write. In case of xol write we always know that it is code write
>
> Of course it has to, but writing code is the harder of the two
> problems. With writes to data segments, the only thing that has to
> be dealt with is the data cache. With code, not only do you need to
> deal with the data cache, but you also need to deal with the instruction
> cache too.
>
>> 2) as I pointed before flush_ptrace_access handles smp case whereas
>> xol write does not need to do that
>
> Are you sure about that?
>
> If I'm reading the code correctly, uprobes inserts a trapping instruction
> into the userspace program. When that instruction is hit, it checks
> whether the thread is the desired one, and may request a slot in this
> magic page, which is when the write happens.
>
> The uprobes special page is shared across all threads which share the
> mm_struct, so in the case of a multi-threaded program running on a SMP
> machine, this page is visible to multiple CPUs.
>
> Is it possible for uprobes to be active on more than one thread at a
> time? If so, because that page is shared, you could end up writing
> to a partial cache line from two threads. From what I can see, ixol[]
> is two words, and there's normally 8 works per cache line on ARM, or
> occasionally 16.
>
> So, the question now is: is it possible to have uprobes active on more
> than one thread, and for two threads to hit the uprobes processing, both
> needing a slot in the page, hitting the same cache line?
>
> Now, what happens if thread 1 on CPU1 gets there first with its write.
> Then thread 2 on CPU2 gets there, causing the cache line to migrate to
> CPU2. Then CPU1 does it's (non-broadcasted) flush, meanwhile CPU2 then
> gets preempted and goes off and does something else.
>
> Please tell me that can't happen. :)
>
From arch/arm/include/asm/uprobes.h:
3) flush_ptra#define UPROBE_XOL_SLOT_BYTES 64
-dl
^ permalink raw reply [flat|nested] 17+ messages in thread
* [RFC PATCH] ARM: uprobes need icache flush after xol write
2014-04-08 16:19 ` Russell King - ARM Linux
2014-04-08 16:29 ` David Long
@ 2014-04-08 18:39 ` Victor Kamensky
1 sibling, 0 replies; 17+ messages in thread
From: Victor Kamensky @ 2014-04-08 18:39 UTC (permalink / raw
To: linux-arm-kernel
On 8 April 2014 09:19, Russell King - ARM Linux <linux@arm.linux.org.uk> wrote:
> On Tue, Apr 08, 2014 at 08:35:01AM -0700, Victor Kamensky wrote:
>> Looking at flush_ptrace_access more closely. Now I am not sure that
>> ptrace write code could easily reused.
>>
>> 1) flush_ptrace_access seems to handle both data and text segments
>> write. In case of xol write we always know that it is code write
>
> Of course it has to, but writing code is the harder of the two
> problems. With writes to data segments, the only thing that has to
> be dealt with is the data cache. With code, not only do you need to
> deal with the data cache, but you also need to deal with the instruction
> cache too.
>
>> 2) as I pointed before flush_ptrace_access handles smp case whereas
>> xol write does not need to do that
>
> Are you sure about that?
>
> If I'm reading the code correctly, uprobes inserts a trapping instruction
> into the userspace program. When that instruction is hit, it checks
> whether the thread is the desired one, and may request a slot in this
> magic page, which is when the write happens.
>
> The uprobes special page is shared across all threads which share the
> mm_struct, so in the case of a multi-threaded program running on a SMP
> machine, this page is visible to multiple CPUs.
>
> Is it possible for uprobes to be active on more than one thread at a
> time? If so, because that page is shared, you could end up writing
> to a partial cache line from two threads. From what I can see, ixol[]
> is two words, and there's normally 8 works per cache line on ARM, or
> occasionally 16.
>
> So, the question now is: is it possible to have uprobes active on more
> than one thread, and for two threads to hit the uprobes processing, both
> needing a slot in the page, hitting the same cache line?
>
> Now, what happens if thread 1 on CPU1 gets there first with its write.
> Then thread 2 on CPU2 gets there, causing the cache line to migrate to
> CPU2. Then CPU1 does it's (non-broadcasted) flush, meanwhile CPU2 then
> gets preempted and goes off and does something else.
>
> Please tell me that can't happen. :)
>
>> 3) flush_ptrace_access needs vma, which uprobe code does not have on
>> layer where xol is installed. That is probably most critical issue that
>> could stop suggested code sharing. And note that vma in
>> flush_ptrace_access is needed only for cases cases 1) and 2) above,
>> about which xol write does not really care. If we take only required
>> pieces from flush_ptrace_access would not xol cache flush look like
>> this:
>>
>> void arch_uprobe_xol_sync_dcache_icache(struct page *page,
>> unsigned long addr, unsigned long size)
>> {
>> if (icache_is_vipt_aliasing())
>> flush_icache_alias(page_to_pfn(page), addr, size);
>> else
>> __cpuc_coherent_user_range(addr, size);
>> }
>
> And this is only half the story.
>
> What you're saying above is:
>
> - if the *instruction* cache is an aliasing VIPT cache, then we can
> map the user page at the same colour and flush the data and instruction
> caches according to that colour.
> Problem: if you have a VIPT aliasing data cache, and you wrote
> to the page via a mapping which had a different colour
> (which is highly likely given you're using kmap_atomic())
> then you are not flushing the new instructions out of
> the data cache.
> - if the *instruction* cache is not an aliasing VIPT cache, then we
> flush using the userspace mapping directly.
> Problem: what if the data cache is aliasing... same problem
> as the above case.
>
> This is why we have the check for cache_is_vipt_aliasing() in there -
> that's not _just_ to deal with ptrace data modification, it's there for
> text modification as well.
Thanks! I got it now. Did not fully understand flush_ptrace_access code.
And did not catch (misread) difference between cache_is_vipt_aliasing
and icache_is_vipt_aliasing().
Probably some technique that retrieves required information from
vma like
1) is_exec = vma->vm_flags & VM_EXEC
2) core_in_mm = cpumask_test_cpu(smp_processor_id(), mm_cpumask(vma->vm_mm))
3) broadcast = cache_ops_need_broadcast()
and passes it to common function would allow to share common
code and relieve need to search for vma after xol write call. After
xol write will call common function will be called with is_exec = 1,
core_in_mm = 1, broadcast = 0.
Although that common function will have seven parameters and
will look a bit ugly and unnatural. It could be compensated by
making it inline so flush_ptrace_access and flush_uprobe_xol_access
function would effectively have its own copy and correctly optimized.
Other option is to have flush_ptrace_access that takes vma and
flush_uprobe_xol_access function side by side in the same file
with comment that begs to keep common logic between two functions.
But not sure how that would work in long term.
Comments? Preferences? Any other suggestions how to deal with
lack of vma in uprobe xol write case?
Thanks,
Victor
>> Note on the side: flush_ptrace_access works with user-land memory
>> but it still uses __cpuc_coherent_kern_range. I think it should use
>> __cpuc_coherent_user_range.
>
> No, because __cpuc_coherent_kern_range() there is used on the *kernel*
> mapping of the page, not on the *user* mapping (which may fault). The
> kernel mapping will never fault.
>
>> It looks like implementation wise it is
>> the same but user variant seems more appropriate in given situation.
>> Or am I missing something here?
>
> They do seem to be the same for both cases, so we could probably remove
> the distinction. It's unlikely we'll ever see an implementation needing
> a difference between the two now.
>
> --
> FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
> improving, and getting towards what was expected from it.
^ permalink raw reply [flat|nested] 17+ messages in thread
* [RFC PATCH] ARM: uprobes need icache flush after xol write
2014-04-08 15:41 ` Russell King - ARM Linux
@ 2014-04-09 16:18 ` Russell King - ARM Linux
2014-04-09 16:38 ` Russell King - ARM Linux
2014-04-09 18:24 ` Oleg Nesterov
0 siblings, 2 replies; 17+ messages in thread
From: Russell King - ARM Linux @ 2014-04-09 16:18 UTC (permalink / raw
To: linux-arm-kernel
On Tue, Apr 08, 2014 at 04:41:17PM +0100, Russell King - ARM Linux wrote:
> I think let's start out by killing this function - the semantics of
> this function have been lost, so it's not clear what it was supposed
> to do in its original form. Even going back to the start of git
> history, it looks like it was never used outside arch code.
I floated a patch to remove flush_icache_user_range() to the architecture
maintainers, giving the background to how this came about. I received the
following reply from David Miller:
> ptrace() accesses (via __access_remote_vm()) already use an existing
> helper function for these sorts of situations, in the form of
> copy_{to,from}_user_page(). I would suggest that uprobes uses that
> as well.
I think this is a very valid point, and echo's my point.
--
FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
improving, and getting towards what was expected from it.
^ permalink raw reply [flat|nested] 17+ messages in thread
* [RFC PATCH] ARM: uprobes need icache flush after xol write
2014-04-09 16:18 ` Russell King - ARM Linux
@ 2014-04-09 16:38 ` Russell King - ARM Linux
2014-04-09 18:24 ` Oleg Nesterov
1 sibling, 0 replies; 17+ messages in thread
From: Russell King - ARM Linux @ 2014-04-09 16:38 UTC (permalink / raw
To: linux-arm-kernel
On Wed, Apr 09, 2014 at 05:18:26PM +0100, Russell King - ARM Linux wrote:
> On Tue, Apr 08, 2014 at 04:41:17PM +0100, Russell King - ARM Linux wrote:
> > I think let's start out by killing this function - the semantics of
> > this function have been lost, so it's not clear what it was supposed
> > to do in its original form. Even going back to the start of git
> > history, it looks like it was never used outside arch code.
>
> I floated a patch to remove flush_icache_user_range() to the architecture
> maintainers, giving the background to how this came about. I received the
> following reply from David Miller:
>
> > ptrace() accesses (via __access_remote_vm()) already use an existing
> > helper function for these sorts of situations, in the form of
> > copy_{to,from}_user_page(). I would suggest that uprobes uses that
> > as well.
>
> I think this is a very valid point, and echo's my point.
More comments from David about using copy_{to,from}_user_page():
> They should use it at first, then if there is a problem we can identify
> exactly what can be optimized or needs to be, and expand the interface
> as needed.
>
> Let's not optimize first.
Please look into using this solution, thanks.
--
FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
improving, and getting towards what was expected from it.
^ permalink raw reply [flat|nested] 17+ messages in thread
* [RFC PATCH] ARM: uprobes need icache flush after xol write
2014-04-09 16:18 ` Russell King - ARM Linux
2014-04-09 16:38 ` Russell King - ARM Linux
@ 2014-04-09 18:24 ` Oleg Nesterov
1 sibling, 0 replies; 17+ messages in thread
From: Oleg Nesterov @ 2014-04-09 18:24 UTC (permalink / raw
To: linux-arm-kernel
On 04/09, Russell King - ARM Linux wrote:
>
> I floated a patch to remove flush_icache_user_range() to the architecture
> maintainers, giving the background to how this came about. I received the
> following reply from David Miller:
>
> > ptrace() accesses (via __access_remote_vm()) already use an existing
> > helper function for these sorts of situations, in the form of
> > copy_{to,from}_user_page(). I would suggest that uprobes uses that
> > as well.
>
> I think this is a very valid point, and echo's my point.
Well. So far I disagree, but let me reply tomorrow.
Sorry, right now I can't even read this thread.
Oleg.
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2014-04-09 18:24 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-04-08 3:04 [RFC PATCH] ARM: uprobes need icache flush after xol write Victor Kamensky
2014-04-08 3:04 ` Victor Kamensky
2014-04-08 8:24 ` Dave Martin
2014-04-08 11:46 ` Russell King - ARM Linux
2014-04-08 13:05 ` David Long
2014-04-08 13:30 ` Russell King - ARM Linux
2014-04-08 14:09 ` Victor Kamensky
2014-04-08 15:35 ` Victor Kamensky
2014-04-08 16:19 ` Russell King - ARM Linux
2014-04-08 16:29 ` David Long
2014-04-08 18:39 ` Victor Kamensky
2014-04-08 15:27 ` Oleg Nesterov
2014-04-08 15:41 ` Russell King - ARM Linux
2014-04-09 16:18 ` Russell King - ARM Linux
2014-04-09 16:38 ` Russell King - ARM Linux
2014-04-09 18:24 ` Oleg Nesterov
2014-04-08 14:15 ` Victor Kamensky
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.