Power To The People!

reddy kilowattOne of the most challenging problems faced by field service techs and engineers is intermittent failures that appear to have no discernible root cause.   Well…as least not an obvious one like a loose cable or belt.   More often than not, the most common cause of such failures is overlooked…electrical power.    Incoming AC (alternating current) power has been around so long that most of us take it for granted.  So long as the lights come on, we assume all is well.    That’s not always true.

In North America, most labs have either 110VAC  (actually somewhere between 105-125VAC) or 240VAC (for larger instruments like freezers or floor mount centrifuges). The AC power that feeds most lab instruments is converted into DC power via the instruments internal power supply which also steps it down to power integrated circuits, dc motors, relays, solenoids..etc (generally in the 5-24VDC range).    Power supplies are pretty robust devices that can provide constant, clean DC voltage, but like many things in life the quality of the output is a function of the quality of the input.  Garbage in = Garbage Out.

Unlike DC voltage which if looked at with an oscilloscope would show a flat line,  AC voltage is actually a sine wave and in most cases, the rated voltage of a circuit is represented by the avevoltagerage (RMS) of the voltages under the curve over time (usually 50 or 60 HZ or cycles per minute).   Now, garbage might be a harsh term but what we are really talking about are several common problems;

Voltage Spikes – Sometimes called a surge, a result of incoming voltage exceeding the rated voltage by 10% or more.   This typically happens when an inductive load (like a centrifuge) is turned off.   The centrifuge pulls a lot of current  and taking that load away (current) allows the voltage to quickly increase (ie, spike).   If an instrument has a well designed regulated power supply then no problem, but transient spikes (think lightening) have been known to take out the best designed power supplies.

Voltage Dips – a temporary drop of  more than 10%  (ex: 120V * .9 = 108V).  Probably not a killer, but what if your device is spec’d at 120V and the input line power is only 110VAC?  Now with a voltage dip you are talking 99VAC …  Some instrument power supplies have the ability to detect under voltages and report errors, many do not.   And…guess what usually happens after a dip?   You guessed it, a spike!

Noise – AC-powered devices can create a characteristic hum at  multiples of the frequencies of the AC power that they use.  Hums are commonly produced by spinning motor and transformer core laminations vibrating in time with the magnetic field.   The noises can wreak havoc on under-voltage situations as they can temporarily cause an instrument on the hairy edge to work temporarily.  Shut the noisy device off and the line dips down again causing the instrument to fail (or act really weird).

What to do if you suspect power issues?  Well for starters, whenever an instrument starts to show ‘random’ failures;

  • Have facilities verify incoming power.  A digital voltmeter can be used for this, but make sure they are using the RMS (root, mean, square) setting to capture the average voltage.   When in doubt, put a scope on it.  Scopes can also show noise as well as nominal voltages.
  • Isolate the instrument in question.  Make sure there are no other devices on the same circuit.
  • Put a digital or analog chart recorder on the circuit and monitor the line over chart recorderseveral days. Sometimes called a strip recorder, the analog version looks like the lie detectors you see in crime shows.  A needle draws on the paper producing peaks whenever it sees a spike or dip.  Newer digital units do the same thing but are much less intimidating to  less truthful members of society…
  • Note the time that failures occur.   Not surprisingly, spikes and dips tend to occur in larger facilities when people arrive at work, go to lunch, take breaks or go home around the same times.   PC’s , HVAC,  lights are turned on/ off – all in the name of conservation…the laws on unintended consequences.
  • Plug the instrument into  a line conditioner, then plug the conditioner into the circuit.  Power conditioner are good for removing noise and higher-end models (don’t go to Home Depot for this…) offer some protection for spikes and under-voltages.   Not to be confused with Un-interruptable Power Supplies (UPS) which offer a measure of time insurance in the event of a total power loss.   Even basic UPS’ are available with spike and dip protection but even without they are still not a bad investment if you have dirty or unreliable incoming power and no easy way to fix it…

Net/net – don’t be so quick to blame an instrument for abhorrent behavior.   Sometimes it is best to recall a bit of Shakespeare…”the fault dear Brutus lies not in our stars, but im ourselves (or our facilities).”

Posted in Lab Maintenance | Tagged , , , , , , , , , | Leave a comment

Good Reads about Mult-Vendor Support

support

Thought I would share a few articles and interviews that talk about Asset Management and Multi-Vendor Service support.

Next Generation PharmaceuticalOutsourcing Asset Management,  Bob Moore – GE Healthcare, interview

Lab Manager MagazineThe Evolving Service Model   ; Good overview of service offerings from GE, Agilent, PE and Thermo Fisher.

BioScience Technology.comManaging More Lab Assets

GENLab-Asset Management Gets Smarter; older article (circa 2008) but shows that Asset Mgt within life sciences has been around for awhile.

Posted in Lab Maintenance | Tagged , , , , , , , | Leave a comment

Separation Anxiety?

Most labs have used floor mount or bench top centrifuges for separation based assays for decades.  Whether spinning samples to remove air bubbles, spinning down cellular debris or isolating supernatent, there are numerous manual access centrifuges on the market, but when it comes to automation, the choices are limited.

For a number of years, Agilent (formerly Velocity11) has offered the compact VSpin.  VSpin has a two position rotor with buckets for std microplates.   It can spiV11_prod_big_vspinn up to 300o rpm/ 1000g and has an automated door that allows direct access to plates using an offset robot gripper.   Units can be stacked on top of each other for increased  use of vertical workspace.  The Optional Access2 loader can also grab the plate and present it externally to a liquid handler gripper or top loading plate mover like Twister2 or KiNEDx.

Hettich also provides a larger unit called the Rotanta 460 which can accommodate 4 plates at speeds up to Hettich_Rotanta_46_RSC_Front_Hatch6200prm,  but is a bit more of a challenge to integrate as the robot gripper fingers need to reach into the unit from the top.  I have seen this done with Mitsubishi and Staubli robots and Tecan actually integrates this unit under an EVO liquid handler accessible via an open locator in the deck.Ixion3

Sias’s Ixion is a compact unit, similar in size to the VSpin, however plate access (total of two) is through the top just like the Rotanta and can spin up to 2000rpm.   This unit integrates nicely with Sias’ Xantus liquid handlers.

Finally, BioNex offers the HiG centrifuge which can also spin two plates.  The bright orange color makes this unit hard to ignore…and a closer look shows that this unit may be BioNex HiGthe best of the bunch.   With an automated lid that retracts from the top, the HiG does not need a plate loader like the VSpin as plates can be accessed by just about any robot gripper.   At 5000g, BioNex claims this unit to be the fastest robot accessible centrifuge available.

Maintenance requirements for each of these devices is similar.   All include high-speed motors so proper ventilation is a must.   Bearings must be greased, sensors cleaned and pneumatics (door opening, plate loaders) checked for leaks.   Additionally, rotors and buckets should be checked for cracks or other signs of wear.   As noted in previous blogs, rotational speeds can be verified using a digital tachometer but you may need to remove covers to gain access to the rotor (kids, don’t try this at home…call a professional).   As always, if you ignore that last piece of advice, don’t come crying to me when your friends make fun of you because you have a mircrotitre plate permanently embedded in your cheek…

Posted in Centrifuges, integrated system, Lab Maintenance, Liquid Handlers, Multi Vendor Services | Tagged , , , , , , , , , , , | Leave a comment

Every Picture Tells A Story, Don’t It?

A picture is worth a thousand words…so even at a reduced frame rate of 15FPS, one minute of video has to be worth 900,00 words.” – Me

For better or worse, advances in cellular communications are cameraphonemaking the once seemly impossible, trivial.  Specifically, I am referring to video communication.   Just about everyone has a ‘smartphone’ these days and it is hard to find a new phone that does not include a camera.   The resolution of these cameras is incredible (the Apple iPhone 5 = 8 MegaPixels) and product stunningly clear videos and images.

Video applications such as Apple’s FaceTime and Skype make face to face remote communications simple, fast and cheap.   For service organizations, this has provided thermal imagingfield based techs with an incredible tool for diagnosing instrument failures.    There are even iPhone apps that now allow users to perform thermal imaging (how cool is that…no pun intended)!  Let’s face it, the pressure these on-site techs feel when faced with a failed instrument can be enormous.  End user anxiety and a ticking clock only add to the stress.   The ability to ‘phone a friend’, point the phone at the instrument and have a real-time conversation about such failures brings an added dimension to peer review.

On the wired side, I have visited many research labs that have added low-cost USB or Ethernet cameras to their automation systems that allow them to monitor status remotely (many times from home, over a weekend or at night).   When combined with remote network access tools like PC Anywhere or LogMeIn, it is possible to deal with simple application errors and continue assays or applications that would otherwise had to wait for human to come into the lab and simple press a key.   Remote observation in this fashion requires network access and must always include IT departments to prevent unauthorized access.

Still, many labs will not allow non-employee cameras or video use within their labs.  This skype5is short-sighted (IMO), and unfortunate.   I understand the competitive nature of pharmaceutical or biotech research and the commercial implications of potentially providing competitors with a glimpse of a labs inner workings, but let’s face it…it would take a pretty savvy bunch of people to gleam something worthwhile from a phone camera.  Instrument failures that render an instrument ‘down’ are generally easier to diagnose and repair, however it the aberrant or irregular failures that could benefit immensely from remote observation.   Unless an instrument or system is under a service contract it can be very expensive to pay for a service tech to sit and watch for a reported failure (they always happen when the tech leaves, right?).

Most labs require non-disclosure agreements or safety training prior to granting non-employees access their labs and the time is well past to include the use of remote diagnostic tools, particularly cellular video in such protocols.   Perhaps seeing is believing?

Posted in integrated system, Lab Maintenance, Multi Vendor Services | Tagged , , , , , , , | Leave a comment

What is the opposite of TMI?

I am a big fan of Lab Manager Magazine.   I am an on-line and print subscriber anlab managerd find it to be a great source of information regarding lab trends and support.

Having said that…I was a bit disappointed by a recent “Ask The Expert” interview by Tanuja Koppal, PhD.  It was called “Optimizing Lab Services: Evaluating the Single-Vendor Option.”   You can read the full article by clicking here.

Although there are some good insights there were some major pieces of informationthe-godfather-brando-150x150 missing.  For starters, it does not mention who the subject of the interview is.  I will give Dr. Koppal the benefit of the doubt and assume the interviewee is not fictitious, but I have a hard time understanding why he/she would need to anonymized.   Is there an MVS Mafia out there that requires a witness protection program?   Secondly, all the MVS providers whom the user evaluated are also anonymized.   I guess I could understand that given that many of these larger providers may have legal teams that would give any crime syndicate a scare.

In the spirit of peer review, I think it would be extremely helpful to both MVS providers and potential customers to know who this customer is and how they made the selection they did.

Who knows, using this feedback, maybe next time they need a contract, someone would be able to make them an offer they couldn’t refuse…

Posted in Lab Maintenance, Multi Vendor Services, Uncategorized | Tagged , , , , , , , , | Leave a comment

Well Equipped…Part II

WD40 Duct Tape Flow Chart“One only needs two tools in life WD-40 to make things go, and duct tape to make them stop.” –  G. Weilacher

While true in many facets of instrument support, there is one other tool which lab support techs will find invaluable – the digital tachometer.    A more precise name would be a strobe tachometer, which neatly describes the basic theory of operation…a strobe light which is used to monitor the rotational speed of the rotor.tach

The LW Scientific Hand-Held tachometer can be found at a variety of web stores for under $200.   This device is easy to use and comes with reflective tape targets that you can place on the rotor arm.  Simply point the tach at the rotor in the general area of the target and hold it steady…after a few seconds you will see a reading that while changing, stays within the commanded speed.   This device can monitor speeds from 20-50,000 RPM which makes it ideal for most lab centrifuges and has a range of 50-400mm.  It’s accurate to +/- 20 RPM, so obviously you would want to be a bit skeptical at the low end range…

Interestingly, most separation assays call for acceleration of the sample not the rotational speed.  From a repair or assay integrity perspective, checking RPM’s will suffice as a general method to determine that the instrument is performing as specified by the manufacturer.    For those who are more curious, there is a great Wiki with more info.

Some centrifuge brands have sight glass windows that allow the digital tach to observe rotor speed while the unit is running…others do not.  Now comes the inevitable caution…caution! (notice I even used an exclamation point).    Seriously, most centrifuges Beck_L8are capable of causing great physical harm due to their extremely high speeds.   Safety interlocks that prevent internal access while spinning are there for a reason.   While a trained tech can defeat such locks, it is not advisable for a novice.   If you any doubts click this image to learn how dangerous high-speed centrifuges can be…

Manufacturer or third-party FSE’s re-calibrate the speed of a centrifuge by adjusting one or more potentiometers on the control board.  Initial speed setting is typically done without a rotor in the unit.

One last caution kids…speed kills.   Let’s be careful out there.

Posted in Centrifuges, Lab Maintenance | Tagged , , , , , , , | Leave a comment

Well Equipped…Part 1

Not every instrument failure requires a call to the manufacturer (OEM)  or an independent service organization (ISO).   Some simple and common failures can be rectified by just about anyone with some common sense and common tools.      Can’t help much with the common sense, but the tool part is a lot more straight forward.

***WARNING *** if you are not comfortable working with electricity please don’t mess around and call for help from you own facilities support folkdmms or and ISO.    If you kill yourself, don’t write me a nasty-gram from the afterlife.

The handheld DMM –  Digital Multimeter (aka the voltmeter).   The name voltmeter is used pretty loosely by a lot of tech’s and only describes one function of this device.  Very capable DMM’s can be found at the local hardware store for under US$50.     For a good tutorial click here.

Voltage – Most DMM’s can measure a wide range of AC or DC voltage.   One of the most common problems when you fire up an instrument and get nothing is no AC power.   Most US labs will operator on 110 or 200VAC.    A zero volt reading means you probably have popped a circuit breaker.  If the AC outlet you are plugged into had a ground fault button, try pressing the reset button and try again.   If you get voltage at the outlet, but no action on the instrument, you may have blown a fuse.  Not comfortable checking voltage?  Try plugging the instrument into a known good working outlet instead.   More knowledgeable techs can test DC voltages for printed circuit boards (PCB’s) inside the instrument.  Most instrument power supplies will convert AC power into lower voltage DC power and distribute it throughout the instruments.    Many PCB’s have incoming power marked at a connector coming from the main power supply.

Resistance –  Resistance is a measure of a devices ability to restrict the flow of electrons in a circuit.   If you crack open an instrument and see a charred component, it is likely a burned out resistor.   If you can still see the value of that resistor  (some have the value printed, others may use a series of colored bands), you can use the DMresistorM to verify if it is blown (open circuit, infinite resistance).   While you may be able to unsolder and replace this component, there is no guarantee that it will not blow again, as something else may have failed that caused too much current to flow thru it or too much voltage across it, causing it to cook.  If you come across a cooked resistor (or any other component), better to have someone replace the entire module.   Almost no FSE’s will spend time doing component level failure analysis as it is time consuming and ultimately more expensive.

Continuity – Some DMM’s allow you test for continuity (the closure of a circuit) that will result in a beeping signal.   No beep, no continuity.   A quick crossing of the probe leads will tell you what sound you are listening for.   This is what you will use to check you fuses or diode.    A diode allows current to flow in one direction only.   Diodes can be checked by reversing the leads across the component.  It should beep with the leads in one positreceptacleion, not beep in the other.  Some instruments have a main fuse as part of the receptacle that the AC cord plugs into.   MAKE SURE YOU UNPLUG THE INSTRUMENT BEFORE YOU DO THIS!!!   You can pop this open and check if the fuse is good or not.

Current –  Not really something I would advise a notice to attempt.   While voltage is measured across a load, current is measured in series with a load.   So, in order to check current, you need to break the circuit and use the DMM to measure current flowing through the meter as part of the circuit.   Lots of potential to hurt yourself here…leave to a professional.

Temperature – One of the features of many digital mulitmeters versus their older analog counterparts is the inclusion of a thermometer probe.   This can be very hand for diagnosing random failures that are related to run away heating problems –  a common example might be an intermittent cooling fan failure.   Try taking a cover off near the fan, tape the probe somwhere close and note the temperature during normal operation with fan running (and cover back on).   Then open it up, and unplug the fan (replace the cover) and monitor the temperature increase.  If you do this, be vigilant and don’t leave the instrument unattended.   You are looking not only for a temperature spike but also abhorrent instrument behavior…so you want to be able to shut it down ASAP.electicution

Okay, so there you have it.  Some basic things you can do with a DMM.  Just remember, when it comes to anything involving electricity, you should always consult with your facilities management.   Never perform electrical testing alone and never in the presence of liquids (especially flammables).   When in doubt…leave to someone in the know.

Posted in Lab Maintenance | Tagged , , , , , | Leave a comment