libsas.rst 13.7 KB
Newer Older
1
2
3
.. SPDX-License-Identifier: GPL-2.0

=========
4
SAS Layer
5
=========
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

The SAS Layer is a management infrastructure which manages
SAS LLDDs.  It sits between SCSI Core and SAS LLDDs.  The
layout is as follows: while SCSI Core is concerned with
SAM/SPC issues, and a SAS LLDD+sequencer is concerned with
phy/OOB/link management, the SAS layer is concerned with:

      * SAS Phy/Port/HA event management (LLDD generates,
        SAS Layer processes),
      * SAS Port management (creation/destruction),
      * SAS Domain discovery and revalidation,
      * SAS Domain device management,
      * SCSI Host registration/unregistration,
      * Device registration with SCSI Core (SAS) or libata
        (SATA), and
      * Expander management and exporting expander control
        to user space.

A SAS LLDD is a PCI device driver.  It is concerned with
phy/OOB management, and vendor specific tasks and generates
events to the SAS layer.

The SAS Layer does most SAS tasks as outlined in the SAS 1.1
spec.

The sas_ha_struct describes the SAS LLDD to the SAS layer.
Most of it is used by the SAS Layer but a few fields need to
be initialized by the LLDDs.

After initializing your hardware, from the probe() function
you call sas_register_ha(). It will register your LLDD with
the SCSI subsystem, creating a SCSI host and it will
register your SAS driver with the sysfs SAS tree it creates.
It will then return.  Then you enable your phys to actually
start OOB (at which point your driver will start calling the
notify_* event callbacks).

43
44
45
46
47
Structure descriptions
======================

``struct sas_phy``
------------------
48
49

Normally this is statically embedded to your driver's
50
51
52
53
54
55
56
57
phy structure::

    struct my_phy {
	    blah;
	    struct sas_phy sas_phy;
	    bleh;
    };

58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
And then all the phys are an array of my_phy in your HA
struct (shown below).

Then as you go along and initialize your phys you also
initialize the sas_phy struct, along with your own
phy structure.

In general, the phys are managed by the LLDD and the ports
are managed by the SAS layer.  So the phys are initialized
and updated by the LLDD and the ports are initialized and
updated by the SAS layer.

There is a scheme where the LLDD can RW certain fields,
and the SAS layer can only read such ones, and vice versa.
The idea is to avoid unnecessary locking.

74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
enabled
    - must be set (0/1)

id
    - must be set [0,MAX_PHYS)]

class, proto, type, role, oob_mode, linkrate
    - must be set

oob_mode
    - you set this when OOB has finished and then notify
      the SAS Layer.

sas_addr
    - this normally points to an array holding the sas
      address of the phy, possibly somewhere in your my_phy
      struct.

attached_sas_addr
    - set this when you (LLDD) receive an
      IDENTIFY frame or a FIS frame, _before_ notifying the SAS
      layer.  The idea is that sometimes the LLDD may want to fake
      or provide a different SAS address on that phy/port and this
      allows it to do this.  At best you should copy the sas
      address from the IDENTIFY frame or maybe generate a SAS
      address for SATA directly attached devices.  The Discover
      process may later change this.

frame_rcvd
    - this is where you copy the IDENTIFY/FIS frame
      when you get it; you lock, copy, set frame_rcvd_size and
      unlock the lock, and then call the event.  It is a pointer
      since there's no way to know your hw frame size _exactly_,
      so you define the actual array in your phy struct and let
      this pointer point to it.  You copy the frame from your
      DMAable memory to that area holding the lock.

sas_prim
    - this is where primitives go when they're
      received.  See sas.h. Grab the lock, set the primitive,
      release the lock, notify.

port
    - this points to the sas_port if the phy belongs
      to a port -- the LLDD only reads this. It points to the
      sas_port this phy is part of.  Set by the SAS Layer.

ha
    - may be set; the SAS layer sets it anyway.

lldd_phy
    - you should set this to point to your phy so you
      can find your way around faster when the SAS layer calls one
      of your callbacks and passes you a phy.  If the sas_phy is
      embedded you can also use container_of -- whatever you
      prefer.


``struct sas_port``
-------------------

135
136
137
138
139
140
The LLDD doesn't set any fields of this struct -- it only
reads them.  They should be self explanatory.

phy_mask is 32 bit, this should be enough for now, as I
haven't heard of a HA having more than 8 phys.

141
142
143
144
lldd_port
    - I haven't found use for that -- maybe other
      LLDD who wish to have internal port representation can make
      use of this.
145

146
147
``struct sas_ha_struct``
------------------------
148
149

It normally is statically declared in your own LLDD
150
151
152
153
154
155
156
157
158
structure describing your adapter::

    struct my_sas_ha {
	blah;
	struct sas_ha_struct sas_ha;
	struct my_phy phys[MAX_PHYS];
	struct sas_port sas_ports[MAX_PHYS]; /* (1) */
	bleh;
    };
159

160
    (1) If your LLDD doesn't have its own port representation.
161
162
163
164

What needs to be initialized (sample function given below).

pcidev
165
166
167
168
^^^^^^

sas_addr
       - since the SAS layer doesn't want to mess with
169
170
171
172
	 memory allocation, etc, this points to statically
	 allocated array somewhere (say in your host adapter
	 structure) and holds the SAS address of the host
	 adapter as given by you or the manufacturer, etc.
173

174
sas_port
175
176
177
178
^^^^^^^^

sas_phy
      - an array of pointers to structures. (see
179
180
	note above on sas_addr).
	These must be set.  See more notes below.
181
182
183

num_phys
       - the number of phys present in the sas_phy array,
184
185
186
187
188
	 and the number of ports present in the sas_port
	 array.  There can be a maximum num_phys ports (one per
	 port) so we drop the num_ports, and only use
	 num_phys.

189
The event interface::
190
191

	/* LLDD calls these to notify the class of an event. */
192
193
194
195
	void sas_notify_port_event(struct sas_phy *, enum port_event);
	void sas_notify_phy_event(struct sas_phy *, enum phy_event);
	void sas_notify_port_event_gfp(struct sas_phy *, enum port_event, gfp_t);
	void sas_notify_phy_event_gfp(struct sas_phy *, enum phy_event, gfp_t);
196

197
The port notification::
198
199
200
201
202
203
204
205
206

	/* The class calls these to notify the LLDD of an event. */
	void (*lldd_port_formed)(struct sas_phy *);
	void (*lldd_port_deformed)(struct sas_phy *);

If the LLDD wants notification when a port has been formed
or deformed it sets those to a function satisfying the type.

A SAS LLDD should also implement at least one of the Task
207
Management Functions (TMFs) described in SAM::
208
209
210
211
212
213
214
215
216
217
218
219

	/* Task Management Functions. Must be called from process context. */
	int (*lldd_abort_task)(struct sas_task *);
	int (*lldd_abort_task_set)(struct domain_device *, u8 *lun);
	int (*lldd_clear_aca)(struct domain_device *, u8 *lun);
	int (*lldd_clear_task_set)(struct domain_device *, u8 *lun);
	int (*lldd_I_T_nexus_reset)(struct domain_device *);
	int (*lldd_lu_reset)(struct domain_device *, u8 *lun);
	int (*lldd_query_task)(struct sas_task *);

For more information please read SAM from T10.org.

220
Port and Adapter management::
221
222
223
224
225
226
227

	/* Port and Adapter management */
	int (*lldd_clear_nexus_port)(struct sas_port *);
	int (*lldd_clear_nexus_ha)(struct sas_ha_struct *);

A SAS LLDD should implement at least one of those.

228
Phy management::
229
230
231
232

	/* Phy management */
	int (*lldd_control_phy)(struct sas_phy *, enum phy_func);

233
234
235
lldd_ha
    - set this to point to your HA struct. You can also
      use container_of if you embedded it as shown above.
236
237
238

A sample initialization and registration function
can look like this (called last thing from probe())
239
*but* before you enable the phys to do OOB::
240

241
242
243
244
245
    static int register_sas_ha(struct my_sas_ha *my_ha)
    {
	    int i;
	    static struct sas_phy   *sas_phys[MAX_PHYS];
	    static struct sas_port  *sas_ports[MAX_PHYS];
246

247
	    my_ha->sas_ha.sas_addr = &my_ha->sas_addr[0];
248

249
250
251
252
	    for (i = 0; i < MAX_PHYS; i++) {
		    sas_phys[i] = &my_ha->phys[i].sas_phy;
		    sas_ports[i] = &my_ha->sas_ports[i];
	    }
253

254
255
256
	    my_ha->sas_ha.sas_phy  = sas_phys;
	    my_ha->sas_ha.sas_port = sas_ports;
	    my_ha->sas_ha.num_phys = MAX_PHYS;
257

258
	    my_ha->sas_ha.lldd_port_formed = my_port_formed;
259

260
261
	    my_ha->sas_ha.lldd_dev_found = my_dev_found;
	    my_ha->sas_ha.lldd_dev_gone = my_dev_gone;
262

263
	    my_ha->sas_ha.lldd_execute_task = my_execute_task;
264

265
266
267
268
269
270
271
	    my_ha->sas_ha.lldd_abort_task     = my_abort_task;
	    my_ha->sas_ha.lldd_abort_task_set = my_abort_task_set;
	    my_ha->sas_ha.lldd_clear_aca      = my_clear_aca;
	    my_ha->sas_ha.lldd_clear_task_set = my_clear_task_set;
	    my_ha->sas_ha.lldd_I_T_nexus_reset= NULL; (2)
	    my_ha->sas_ha.lldd_lu_reset       = my_lu_reset;
	    my_ha->sas_ha.lldd_query_task     = my_query_task;
272

273
274
	    my_ha->sas_ha.lldd_clear_nexus_port = my_clear_nexus_port;
	    my_ha->sas_ha.lldd_clear_nexus_ha = my_clear_nexus_ha;
275

276
	    my_ha->sas_ha.lldd_control_phy = my_control_phy;
277

278
279
	    return sas_register_ha(&my_ha->sas_ha);
    }
280
281
282
283

(2) SAS 1.1 does not define I_T Nexus Reset TMF.

Events
284
======
285

286
Events are **the only way** a SAS LLDD notifies the SAS layer
287
288
289
290
of anything.  There is no other method or way a LLDD to tell
the SAS layer of anything happening internally or in the SAS
domain.

291
292
Phy events::

293
294
295
296
297
	PHYE_LOSS_OF_SIGNAL, (C)
	PHYE_OOB_DONE,
	PHYE_OOB_ERROR,      (C)
	PHYE_SPINUP_HOLD.

298
299
Port events, passed on a _phy_::

300
301
302
303
304
305
306
307
308
309
	PORTE_BYTES_DMAED,      (M)
	PORTE_BROADCAST_RCVD,   (E)
	PORTE_LINK_RESET_ERR,   (C)
	PORTE_TIMER_EVENT,      (C)
	PORTE_HARD_RESET.

Host Adapter event:
	HAE_RESET

A SAS LLDD should be able to generate
310

311
312
313
314
315
316
317
318
	- at least one event from group C (choice),
	- events marked M (mandatory) are mandatory (only one),
	- events marked E (expander) if it wants the SAS layer
	  to handle domain revalidation (only one such).
	- Unmarked events are optional.

Meaning:

319
320
321
322
323
324
325
326
327
328
329
HAE_RESET
    - when your HA got internal error and was reset.

PORTE_BYTES_DMAED
    - on receiving an IDENTIFY/FIS frame

PORTE_BROADCAST_RCVD
    - on receiving a primitive

PORTE_LINK_RESET_ERR
    - timer expired, loss of signal, loss of DWS, etc. [1]_
330

331
332
PORTE_TIMER_EVENT
    - DWS reset timeout timer expired [1]_
333

334
335
PORTE_HARD_RESET
    - Hard Reset primitive received.
336

337
338
PHYE_LOSS_OF_SIGNAL
    - the device is gone [1]_
339

340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
PHYE_OOB_DONE
    - OOB went fine and oob_mode is valid

PHYE_OOB_ERROR
    - Error while doing OOB, the device probably
      got disconnected. [1]_

PHYE_SPINUP_HOLD
    - SATA is present, COMWAKE not sent.

.. [1] should set/clear the appropriate fields in the phy,
       or alternatively call the inlined sas_phy_disconnected()
       which is just a helper, from their tasklet.

The Execute Command SCSI RPC::
355
356
357
358
359
360
361
362
363
364
365
366

	int (*lldd_execute_task)(struct sas_task *, gfp_t gfp_flags);

Used to queue a task to the SAS LLDD.  @task is the task to be executed.
@gfp_mask is the gfp_mask defining the context of the caller.

This function should implement the Execute Command SCSI RPC,

That is, when lldd_execute_task() is called, the command
go out on the transport *immediately*.  There is *no*
queuing of any sort and at any level in a SAS LLDD.

367
Returns:
368

369
370
   * -SAS_QUEUE_FULL, -ENOMEM, nothing was queued;
   * 0, the task(s) were queued.
371

372
373
374
375
376
377
378
379
380
381
382
383
384
385
::

    struct sas_task {
	    dev -- the device this task is destined to
	    task_proto -- _one_ of enum sas_proto
	    scatter -- pointer to scatter gather list array
	    num_scatter -- number of elements in scatter
	    total_xfer_len -- total number of bytes expected to be transferred
	    data_dir -- PCI_DMA_...
	    task_done -- callback when the task has finished execution
    };

Discovery
=========
386
387

The sysfs tree has the following purposes:
388

389
390
391
392
393
394
395
396
    a) It shows you the physical layout of the SAS domain at
       the current time, i.e. how the domain looks in the
       physical world right now.
    b) Shows some device parameters _at_discovery_time_.

This is a link to the tree(1) program, very useful in
viewing the SAS domain:
ftp://mama.indstate.edu/linux/tree/
397

398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
I expect user space applications to actually create a
graphical interface of this.

That is, the sysfs domain tree doesn't show or keep state if
you e.g., change the meaning of the READY LED MEANING
setting, but it does show you the current connection status
of the domain device.

Keeping internal device state changes is responsibility of
upper layers (Command set drivers) and user space.

When a device or devices are unplugged from the domain, this
is reflected in the sysfs tree immediately, and the device(s)
removed from the system.

The structure domain_device describes any device in the SAS
domain.  It is completely managed by the SAS layer.  A task
points to a domain device, this is how the SAS LLDD knows
where to send the task(s) to.  A SAS LLDD only reads the
contents of the domain_device structure, but it never creates
or destroys one.

Expander management from User Space
421
===================================
422
423
424
425
426
427
428
429
430
431
432

In each expander directory in sysfs, there is a file called
"smp_portal".  It is a binary sysfs attribute file, which
implements an SMP portal (Note: this is *NOT* an SMP port),
to which user space applications can send SMP requests and
receive SMP responses.

Functionality is deceptively simple:

1. Build the SMP frame you want to send. The format and layout
   is described in the SAS spec.  Leave the CRC field equal 0.
433

434
open(2)
435

436
2. Open the expander's SMP portal sysfs file in RW mode.
437

438
write(2)
439

440
3. Write the frame you built in 1.
441

442
read(2)
443

444
445
446
4. Read the amount of data you expect to receive for the frame you built.
   If you receive different amount of data you expected to receive,
   then there was some kind of error.
447

448
close(2)
449

450
451
452
453
454
455
456
457
458
459
460
461
462
All this process is shown in detail in the function do_smp_func()
and its callers, in the file "expander_conf.c".

The kernel functionality is implemented in the file
"sas_expander.c".

The program "expander_conf.c" implements this. It takes one
argument, the sysfs file name of the SMP portal to the
expander, and gives expander information, including routing
tables.

The SMP portal gives you complete control of the expander,
so please be careful.