Quantum back-action-evading measurement of motion in a negative mass reference frame

Quantum mechanics dictates that a continuous measurement of the position of an object imposes a random quantum back-action (QBA) perturbation on its momentum. This randomness translates with time into position uncertainty, thus leading to the well known uncertainty on the measurement of motion. As a consequence of this randomness, and in accordance with the Heisenberg uncertainty principle, the QBA puts a limitation—the so-called standard quantum limit—on the precision of sensing of position, velocity and acceleration. Here we show that QBA on a macroscopic mechanical oscillator can be evaded if the measurement of motion is conducted in the reference frame of an atomic spin oscillator. The collective quantum measurement on this hybrid system of two distant and disparate oscillators is performed with light. The mechanical oscillator is a vibrational ‘drum’ mode of a millimetre-sized dielectric membrane, and the spin oscillator is an atomic ensemble in a magnetic field. The spin oriented along the field corresponds to an energetically inverted spin population and realizes a negative-effective-mass oscillator, while the opposite orientation corresponds to an oscillator with positive effective mass. The QBA is suppressed by −1.8 decibels in the negative-mass setting and enhanced by 2.4 decibels in the positive-mass case. This hybrid quantum system paves the way to entanglement generation and distant quantum communication between mechanical and spin systems and to sensing of force, motion and gravity beyond the standard quantum limit.

Continuous measurement of an oscillator position, x(t) =x(0) cos(Ωt) +p(0) sin(Ωt)/(mΩ), where Ω is the frequency and m is the mass, leads to accumulation of the quantum back action (QBA) of the measurement in both the position and momentum,p, non-commuting variables [x,p] = i [1,2]. Measurement QBA was recently observed for a mechanical oscillator [3] and for atomic motion [4]. Suppose, however, that the position is measured relative to an oscillator with a mass m 0 = −m for whichẋ 0 = −p 0 /m. The result of a measurement ofx(t) −x 0 (t) = (x(0) −x 0 (0)) cos(Ωt) + (p(0) +p 0 (0)) sin(Ωt)/(mΩ) depends only on commuting variables, [x −x 0 ,p +p 0 ] = 0. Hence it can be QBA free [5,6] and the uncertainty in the measurement of the relative position (x −x 0 ) 2 can be smaller than the uncertainty x 2 . The first proposal for such a measurement based on atomic spins [6], has been followed by a number of proposals for QBA free measurements [7][8][9][10]. In [11] the negative mass approach referred to as "quantummechanics-free subsystems" was shown to lead to a measurement sensitivity approaching the Cramér-Rao bound. Earlier work on atomic spin ensembles utilized the negative mass property for demonstration of entanglement of macroscopic spins [12] and for entanglement-assisted magnetometry [13]. The back action evading measurement on two mechanical oscillators at room temperature was demonstrated in [14] in the classical regime using light, and recently in the quantum regime at the millikelvin temperature range using microwaves [15]. Ways to overcome QBA limitations for a free mass oscillator with squeezed light have been proposed in [16][17][18].
Here we demonstrate QBA in a novel hybrid quantum system [19,20] composed of a macroscopic mechanical oscillator, a high-Q dielectric membrane [21,22] (Fig. 1a) in a high finesse cavity, and a spin oscillator, an ensemble of room temperature Cesium atoms in a magnetic field contained in a spin-protecting environment [23][24][25] (see Fig. 1b and Supplementary Information). The mechanical oscillator Hamiltonian isĤ M = (mΩ 2 M /2)x 2 + p 2 /2m = ( Ω M /2)(X 2 M +P 2 M ), where we henceforth employ dimensionless variablesX M =x/x zpf andP M = p x zpf / where x zpf = /mΩ M is the oscillator's zero point position fluctuation and [X M ,P M ] = i. Compared to a mechanical oscillator, a spin oscillator has some rather unique properties. Consider a collective atomic spinĴ α = Na i=1F i α with components α = x, y, z composed of a large number N a of ground state spinsF i (with F = 4 in the present case). Atoms are optically pumped to generate an energetically inverted spin population in an external magnetic field B (Fig. 1c), which we take to point in the positive x-direction. The collective spin thus exhibits a large average projection J x = | Ĵ x |/ 1. Its normalized y, z quantum components form canonical oscillator variables [X S , [23] in terms of which the spin Hamiltonian becomeŝ H S = Ω SĴx = Ω S J x − ( Ω S /2)(X 2 S +P 2 S ) with Ω S -the Larmor frequency. The first term is an irrelevant constant energy offset due to the mean spin polarization. The second term is equivalent to the Hamiltonian of a mechanical oscillatorĤ M with a negative mass. Each quantum of excitation in the negative mass spin oscillator physically corresponds to a deexcitation of the inverted spin population by Ω S (Fig. 1c). Preparation of the collective spin in the energetically lowest Zeeman Mechanical and spin oscillators. A. The mechanical oscillator -the (1,2) drum mode, ΩM = 2π×1.28 MHz, of a 0.5 mm, square silicon nitride membrane (light square in the center of the inset) supported by the silicon phononic crystal structure. B. The spin oscillator is an optically pumped gas of Cesium atoms contained in square crossection channel inside a glass cell. Channel walls are coated with a spin-protecting coating. The cell is placed in a static magnetic field with the Larmor frequency ΩS tunable around ΩM . Depending on the direction of the magnetic field with respect to the direction of the atomic spin, the oscillator can have lower (higher) energy of the excited state, corresponding to the negative (positive) effective mass, as shown in C and D, respectively. state realizes instead a positive mass spin oscillator witĥ H S = − Ω S J x + ( Ω S /2)(X 2 S +P 2 S ) (Fig. 1d). The experiment implementing a quantum measurement on the hybrid system is sketched in Fig. 2a, which depicts the cascaded interaction between a traveling light field and the two oscillators (see Supplementary Information for details). A coherent optical field with a strong, classical, linearly polarized component LO 1 (photon flux Φ 1 ) and vacuum quantum fluctuations in the polarization orthogonal to it, described by quadrature phase op-eratorsX L,in andP L,in , first interacts with the spin oscillator. The interaction for far-off-resonant light is of the quantum nondemolition (QND) typeĤ int,S ∝X SXL,in , whereX S ∝Ĵ z is the projection of the collective spin on the direction of light propagation [23]. The light output quadrature,P S L,out (Ω) =P L,in (Ω) + √ Γ SXS (Ω), reads out the atomic spin projectionX S at the rate Γ S ∝ Φ 1 . At the same timeĤ int,S implies that measurement QBA due toX L,in is imprinted on the atomiĉ P S quadrature. The atomic spin projection is driven in addition by intrinsic spin noiseF S so thatX S = Here and henceforth we consider all quantities in Fourier (frequency) domain which is most appropriate for a continuous-time measurement. The atomic oscillator's susceptibility χ S (Ω) = ±2Ω S /(Ω 2 S − Ω 2 − 2iΩγ S ) is determined by the sign of its effective mass (±), resonance frequency Ω S and relax-ation rate γ S (half width at half maximum convention is used throughout the paper). The physics of the QBA in the spin system can be understood as fluctuations of the Stark shift of the atomic energy levels due to fluctuations of the angular momentum of light [23].
The classical drive LO 1 is filtered out after light passes through the atoms (Fig. 2a), whereas the relevant fluctuations in the orthogonal polarization,P S L,out andX S L,out =X L,in are mixed with a classical drive field LO 2 (with photon flux Φ 2 ) in the same polarization and sent onto the mechanical oscillator. The phase of LO 2 is adjusted so thatX M L,in =X S L,out ,P M L,in =P S L,out . The linearized optomechanical Hamiltonian isĤ int,M ∝ X MX M L,in [26]. In analogy with the spin, the output phase quadrature of light,P L,out =P M L,in + √ Γ MXM , reads out the membrane positionX M at the rate Γ M ∝ Φ 2 . The membrane position is driven by thermal state noiseF M and the QBA of light, that isX where the mechanical susceptibility is given by and determined by the mechanical resonance frequency Ω M and damping rate γ M 0 . HenceX L,in is the source of measurement QBA for both the membrane and the spin oscillator.
Overall, the homodyne readout of the joint system with the local oscillator LO 3 can be cast asP L,out = The back action evading character of this measurement comes out most clearly when the measured light quadrature for the joint system is expressed asP L, the terms corresponding to shot noise of light, membrane thermal noise, spin noise, and measurement QBA noise, respectively. Notably, the QBA term shows the interfering responses of the membrane and the spin oscillator. Ideal broadband QBA evasion is achieved for equal readout rates, Γ S = Γ M , and χ M (Ω) = −χ S (Ω) which requires Ω M = Ω S , γ M 0 = γ S and a negative mass spin oscillator (Supplementary Information and [8]).
We exploit the high level of flexibility in our modular hybrid setup to fulfill these requirements: It is straightforward to match the readout rates Γ M Γ S by a proper choice of power levels Φ 1,2 , and to tune the atomic Larmor frequency Ω S to the resonance frequency Ω M = 2π×1.28 MHz of the mechanical drum mode. In order to observe appreciable QBA at the membrane's thermal environment of 7K we use a phononicbandgap shielded membrane with high mechanical quality factor Q corresponding to an intrinsic damping rate of γ M 0 = 2π×50 mHz. On the other hand, the intrinsic spin damping rate γ S0 2π×500 Hz is due to power broadening by optical pumping and atomic collisions. In addition, efficient spin readout requires significant power broadening by the probe light, γ S γ S0 (Supplementary Information), impeding an adjustment of the spin to the  Fig. 2c,d. The experimental parameters are listed in the Extended Data section. Introducing a nonzero detuning also modifies the optomechanical input-output relations and the QBA interference as detailed further below and in the Supplementary Information.
Having matched the susceptibilities and readout rates we perform a back-action limited readout of the two systems as shown in Fig. 2b,c,d. The ratio of QBA from vacuum noise of lightX L,in to thermal noise due toF M (S) is proportional to the quantum cooperativity parameters C M (S) q respectively which we separately calibrate for each system (Supplementary Information). We achieve an optomechanical cooperativity of C M q = 2.5 ± 0.3 and on the side of atoms C S q = 1.10 ± 0.15 which signifies that QBA and thermal noise contribute roughly on the same level in both systems. Fig. 3 displays the results for the hybrid system. As a reference we show the spectra of the two individual systems taken separately (blue -the mechanics, brownthe spin) in Fig. 3a both measured with the LO 3 detector. Fig. 3b presents the hybrid noise for the negative (red) and positive (green) effective spin masses, corresponding to two opposite orientations of the DC magnetic field relative to the spin polarization. The hybrid spectra differ significantly from each other, with the area of the spectrum for the negative (positive) spin mass being significantly smaller (larger) than that for uncorrelated systems -a clear demonstration of the destructive (constructive) interference of the QBA contributions for b a c d the two systems. We emphasize that these data signify a QBA cancellation irrespective of theoretical modelling. For comparison, the Fig. 3a also shows the curve (dashed) obtained by adding the two noise spectra recorded in separate measurements on atoms and the mechanical oscillator. An intriguing feature of the hybrid noise spectra is the apparent absence of interference and noise cancellation exactly at the Fourier frequency Ω = Ω S = Ω M where the negative joint, positive joint and the mechanics spectra overlap (Fig. 3b). This is due to the strong optical broadening of the mechanical oscillator which leads to suppression of the spin phase noise contribution to light on the exact joint resonance. The effect is well understood from the full quantum model (Supplementary Information) and is analogous to optomechanically induced transparency [27]. The solid red, green and blue curves for the negative joint, positive joint and mechanics, respectively, are generated from this model and are in excellent agreement with the data. Fig. 3c presents the spectrum for the hybrid system with the negative mass and the model fit (blue curve) to the spectrum of the mechanics (data in Fig. 3a). The noise reduction of the hybrid spectrum (red dots) compared to the mechanics only (blue curve) in the wings of the spectrum is observable, although its effect is diminished by the added spin thermal noise which is present in the red data, but does not contribute to the blue curve. The observed variance (area) for the joint negative system is 11.2 × x 2 zpf which is (97 ± 2)% of the observed variance for the mechanical oscillator, where the error is derived from the fits. For the positive spin mass the constructive interference of the QBA for the two systems is evident from comparing the green data points to the blue curve for the membrane only (Fig. 3d).
To find the reduction/enhancement of the QBA for the hybrid system, we use the calibration of the thermal noise described in the Supplementary Information and presented in Fig. 2c,d. The mechanical thermal noise found in Fig. 2d is shown as the blue shaded area in Fig. 3c,d. The spin thermal noise found in Fig. 2c is used as an input to the detailed model to find its contribution to the observed hybrid spectra (brown shaded area in Fig. 3c,d). Note that this noise is suppressed by the opto-mechanical response around Ω M = Ω S by the same mechanism as the QBA contribution of the spin is reduced to zero at this point. Subtracting the thermal noise area from the total area, we find the QBA variance contribution for the hybrid negative system of 5.9 × x 2 zpf (striped area in Fig. 3c) and for the hybrid positive system, 11.2 × x 2 zpf (striped area in Fig. 3d). Comparing these values with the QBA of 7.3×x 2 zpf for the mechanical oscillator, we conclude that the variance of the QBA for the joint negative mass system is −1.0 dB (20 ± 5%) be-b a c Figure 4.
Quantum back action for the optimally detuned mechanical and spin oscillators. Noise spectra of detected light. Axes labels: (SN) -shot noise of light, xzpf -zero point fluctuations. A. Membrane noise (blue dots), hybrid system with the negative/positive mass spin oscillator (red/green dots), blue area -membrane thermal noise, brown areaspin thermal noise. Solid brown curve -a fit to the experimental spin spectrum taken without mechanical response. Red area -QBA for the hybrid system. Striped area -QBA for the membrane. B. Displacement sensitivity for the hybrid system with the negative mass (red) normalized to the sensitivity for the mechanical oscillator. C. An example of the squeezed amplitude output of the spin system for positive (black) and negative (brown) effective mass.
low the variance for the mechanics alone, whereas for the joint positive mass system it is 1.7 dB (50 ± 8%) higher. The main contributions to the error bars are the uncertainties in the calibration of quantum cooperativities.
Further studies reveal that a more efficient QBA evasion can be achieved when the two oscillator frequencies are not exactly equal, Ω M = Ω S . Taking advantage of the straightforward tunability of Ω S with magnetic field, we run the QBA evasion experiment with the spin oscillator slightly detuned from the mechanical oscillator. In this case the best QBA evasion is obtained if the quadratures of light between the atomic and the optomechanical systems are rotated with respect to the phase of LO 2 . Fig. 4a shows the data for the hybrid system with the negative spin mass (red dots) with Ω S − Ω M = 2π × 5.2kHz and a phase rotation of 7°, along with the noise of the mechanical oscillator (blue dots). For this experiment we find C M q = 2.2. We observe the broadband QBA evasion which, additionally, is most pronounced at Ω = Ω M where the mechanical response is maximal. The observed total variance for the hybrid system with the negative spin mass, 9.6 × x 2 zpf , is 93 ± 2% of the variance for membrane only, 10.4 × x 2 zpf . Note that interference in the hybrid system leads to suppression of the spin noise (solid brown curve) at Ω S , which is instead transformed into efficient QBA evasion around Ω M for the negative mass hybrid system. Fig. 4b shows the improvement in the membrane displacement sensitivity obtained by the QBA evasion calculated as the ratio of the blue and red curves from Fig. 4a. These data signify broadband QBA evasion in a model independent way.
Subtracting thermal noise contributions we find the hybrid QBA (red area in Fig. 4a) of 4.1 × x 2 zpf , that is −1.7 dB (32 ± 5%) suppression compared to the mechanical QBA of 6.0 × x 2 zpf (striped area). For the hybrid system with the positive spin mass (green dots), the QBA is 10.3 × x 2 zpf which is 2.4 dB (70 ± 10%) above the QBA for the mechanics alone. In this detuned case the QBA reduction in case of negligible thermal noise can, in principle, overcome the limit of 1/2 valid for the case of Ω M = Ω S (see Supplementary Information), as indicated by the 60% reduction of the classical BA that we have observed in an independent experiment with the system driven by classical white noise. The physics of the broadband QBA interference is due to the combination of the frequency dependent amplitude squeezing of the light generated by the spin and the interference of QBA of the two systems. An example of the amplitude squeezed output from the spin in shown in (Fig. 4c).
In conclusion, we have presented a novel hybrid quantum system consisting of distant mechanical and spin oscillators linked by propagating photons. Constructive or destructive interference of the quantum back action for the two oscillators depending on the sign of the effective mass of the spin oscillator is demonstrated. A detailed model describes the results with high accuracy. We have shown that the back action evading measurement in the hybrid system leads to the enhanced sensitivity of the displacement measurement. Further improvements are realistic with reduced propagation losses, even higher Q mechanical oscillators [28] and cavity enhanced spin systems. These results pave the way for entanglement generation and quantum communication between mechanical and spin systems, and to QBA free measurements of acceleration, gravity and force. interface between light and atomic ensembles. Rev. Mod.
Phys. 82, 1041-1093 (2010).  The spin ensemble consists of N a ∼ 10 9 Cesium atoms contained in an anti-relaxation coated pyrex vapor cell (microcell) [S1] heated to a temperature of 65 • C. These atoms are confined in a channel of 300µm × 300µm × 10mm connected to an external Cesium reservoir via a ∼ 10µm radius laser drilled hole, as shown in Figure 1b. They are addressed by light of a waist (radius) size of 55 µm focused through the channel. The microcell is enclosed in a four layer magnetic shielding, protecting the spins from ambient magnetic fields and external RF sources. An inner system of coils produces a homogeneous bias field, B, which leads to a Larmor frequency Ω S . The wall-to-wall transient time of Cs atoms in the channel is on average 1.5 µs. Within the characteristic evolution time of the quantum state, the moving atoms cross the light beam many times and thus experience a motionally averaged interaction [S2, S3]. The spin-protecting coating of the cell walls grants an intrinsic decoherence rate of 300 µs. This rate is limited by spin destruction collisions of atoms with the cell walls, magnetic field inhomogeneities and spin-exchange collisions.
A circularly polarized diode laser tuned to the F = 3 → F = 4 transition of the D2 line is used to spin polarize atoms into the F = 4 ground state. The probing of the atomic ensemble is done with linearly polarized light at 852.3490 nm (LO 1 ), blue detuned by ∆ S ∼ 2π×3 GHz from the F = 4 → F = 5 in the D 2 transition. Absorption effects can be neglected as ∆ S ∆ω HWHM Doppler 2π× 200 MHz, effectively eliminating the effect of spin motion. The polarization of the probe set by the half-waveplate (HWP 0 in Fig. S1) allows for adjustment of the polarization axis of the linearly polarized probing light (LO 1 ). In the experiment this axis is chosen to minimize the added broadening of the spin oscillator, resulting in an angle θ a ≈ 55 • with respect to the direction of the atomic polarization. We emphasize that the dominant part of the light-atoms interaction is of the QND type [S4] (see Eq. (S18)) and does not depend on the angle θ a ; the optical rotation that the light experiences is to a very good approximation independent of the polarization orientation. The vacuum sidebands that affect the spin oscillator are in an orthogonal polarization mode and π/2 out of phase with respect to the local oscillator LO 1 .
The optical readout rate, Γ S ∝ ∆ −2 S Φ 1 |J x |, is a function of the number of atoms J x = F N a , the local oscillator photon flux Φ 1 and the detuning ∆ S . Its origin lies in the re-parametrization of the Faraday rotation experienced by light due to interaction with far detuned atoms [S5, S6]. In the established language of the light-atoms interface [S2], the readout rate is related to the interaction strength κ 2 atoms = Γ S T , with T being the temporal length of the probing light mode. The atomic spins' linewidth, γ S , is dominated by the optical broadening and is proportional to Φ 1 in the regime of interest. Typical values for optical powers and γ S are presented in Table I.
The Cesium spin ensemble fully polarized to F = 4, m F = ±4 has projection noise variance var(Ĵ y,z )  [S7] this is formally equivalent to having a harmonic oscillator with Ω S < 0 (Ω S > 0) as depicted in Figure 1c (Figure 1d) of the main text. Note that throughout the Methods, we include the sign of the effective spin mass in Ω S (whereas this sign is stated as an explicit prefactor in the main text). For the negative (positive) mass oscillator, the first excited state of the spin oscillator is the one with a single atom in the F = 4, m F = 3 (F = 4, m F = −3) state. Experimentally, we change the magnetic field direction to choose the sign of the oscillator's mass.
As presented in Figure 2c, the quantum cooperativity for the spin oscillator, C S q , is characterized via broadband thermal modulation of the optical driving force. This technique requires an electro-optical modulator to drive LO 1 's polarization quadratureX L,in and a RF source outputing thermal voltage noise, producing a frequency independent and proportional to the driving voltage, optical power with excess n WN photon flux in the frequency band of interest. In Figure 2c, the black dots represent the spin oscillator driven with n WN = 1.22. Comparing this curve to the shot-noise limited (n WN = 0) probing, in brown dots, we extract the effect of quantum noise of light on the spin oscillator and the ratio of back action to thermal noise, the quantum cooperativity. For a thorough discussion on this we refer to the Section I G.

B. Optomechanical System
The optomechanical system is based on a near-monolithic cryogenic membrane-in-the-middle system described in detail elsewhere [S8]. The mechanical oscillator is a highly stressed, 60 nm thick SiN membrane supported by a silicon periodic structure forming a phononic bandgap. The bandgap protects the oscillator from phonon tunneling from the clamping of the structure and provides a region clear of undesired phononic modes [S9]. The high stress boosts the quality factor through dissipation dilution [28]. The membrane thickness, t M , is chosen to maximize the optomechanical single photon coupling rate, g 0 . This introduces a tradeoff between the zero point fluctuations and the amplitude reflection coefficient of SiN which is periodic with t M . The (1,2) drum mode of the membrane with frequency Ω M = 2π×1.28 MHz is used as it is the lowest frequency mode to lie within the bandgap and has a high quality factor of Q = 13×10 6 as measured by ring-downs (γ M 0 = 2π×50 mHz). The ∼ 8% side length difference of the membrane breaks the degeneracy of the (1,2) and (2,1) modes significantly, with the (1,2) mode being ∼ 60kHz lower in frequency than its sibling. This membrane is placed in a cavity and aligned such that the cavity TEM 00 mode has a good overlap with the (1,2) mode and a poor overlap with the (2,1) mode. This further separates the systems as the optical spring effect (dynamical back action) pushes the (1,2) mode another ∼ 10kHz away, while having only a marginal impact on the (2,1) mode. The 1.3 mm long plano-concave Fabry-Pérot optical cavity with finesse F = 4500 (half bandwidth of κ = 2π × 13 MHz) is mounted in a continuous flow cryostat with large windows for good optical access and a base temperature of 4.4 K. The power tranmissions of the mirrors are 20 ppm and 1400 ppm thus giving a largely one-sided cavity. Placing the aforementioned dielectric membrane 500 µm from the 20 ppm mirror, forms two sub-cavities whose dynamics can be mapped onto the canonical optomechanical formulation used in the theory section via the transfer matrix model approach described in [S10]. In effect, the cavity half bandwidth κ is modulated depending on the position of the membrane with respect to the standing wave in the cavity [S11]. This is due to a differential intracavity photon population being built up in each sub-cavity. Having more intracavity photons populating the sub-cavity bounded on one side by the low transmission (20 ppm) mirror produces an overall decreased cavity loss rate. In the canonical formulation this is equivalent to the decay rates κ 1 and κ 2 of the cavity ports 1 and 2 being altered asymmetrically, see Fig. S2. The membrane itself adds negligible additional loss.
We position the membrane in the cavity such that the coupling rate is large and where the overall cavity bandwidth is reduced from κ = 2π × 13MHz (bare cavity) to κ = 2π × 7.7MHz (finesse enhanced to 7500). This comes at the expense of having a less one-sided cavity. The ratio of the cavity ports decay rates goes from κ 1 /κ 2 = 70 (bare cavity) to κ 1 /κ 2 = 25. The reduced cavity bandwidth is advantageous as a certain degree of sideband resolution is required to optically broaden the mechanical oscillator significantly without requiring a too large readout rate (required to match the spin system).
The high transmission incoupling mirror is mounted on a piezoelectric transducer, which is used to tune the cavity resonance frequency close to that dictated by the atomic probe LO 1 . It is tuned such that LO 2 probes the cavity red detuned by ∆ = −2π × 4.7 MHz. This is ensured using a separate beam originating from the same laser. This beam is blue shifted (by −∆) from LO 2 described above by an acousto-optic modulator. It is then phase modulated at 12MHz and probes the cavity from the undercoupled port 2, see Fig. S1. An error signal is derived using the Pound-Drever-Hall technique and is used to feedback on the aforementioned piezoelectric transducer which stabilizes the cavity such that this beam is locked on resonance. This locking beam is in the orthogonal polarization to LO 2 and contributes to < 1% of the intracavity power. The locking beam thus has a negligible impact on the intracavity dynamics and the final detection.
The mechanical oscillator is initially only coupled to a thermal bath of temperature T bath with mean occupation n bath . Adding the probe LO 2 alters the dynamics of the system as the oscillator is coupled to the intracavity field with a rate g = g 0 √ N , where N is the mean intracavity photon number. Dynamical back action optically broadens the mechanical linewidth by γ M,opt (such that γ M = γ M 0 + γ M,opt ) and the mean thermal occupation is reduced ton th M = (γ M 0 /γ M )n bath . This (so-called) sideband cooling is due to an asymmetry in the Stokes and anti-Stokes processes caused by the detuning from cavity resonance. The Stokes sideband (causing heating) is never completely suppressed which sets a minimum achievable mean occupationn min M [S12]. The effective mean occupation of the mechanical oscillator is nown M = (γ M 0 /γ M )n bath + (γ M,opt /γ M )n min M , where the contributionn min M is correlated with the quantum back-action.
The total variance of motion will thus have contributions from both the QBA and the thermal bath; the ratio of these is where we have introduced the quantum cooperativity C M q = g 2 0 N/(2κγ M 0nbath ) and approximatedn bath +1/2 ≈n bath (in the present scenarion bath ∼ 10 5 ).
The total variance of the motion can be directly inferred from the area of the measured output spectrum by The approximation is good as the spectra used are dominated by QBA and thermal noise contributions, with a negligible SN contribution, in the frequency range shown in all Figures. By the same token, Eq. (S1) gives operational meaning to the quantum cooperativity C M q as the ratio of QBA and thermal contributions to the observed variance (except for the Lorentzian factor of order unity, 1, seen on the right-hand side).
The optomechanical system is probed in reflection. The combination of a quarter-waveplate at 45 • and a polarizing beam splitter effectively acts as an isolator transmitting the input light (in the hybrid configuration coming from the spin system) and reflecting the light emerging from the cavity. With well characterized optical losses and system parameters, the bath temperature and C M q can be inferred from the observed degree of ponderomotive squeezing. A spectrum (amplitude quadrature) showing squeezing of -1.7 dB (-2.6 dB corrected for detection efficiency of 72%) is shown in Figure 2b. The shot noise (SN) level is verified by balanced detection and by comparison to a white light source to within < 5% accuracy. Using the detailed model (outlined in the theory section) with the bath temperature of the membrane, T bath , as the only adjustable parameter, we obtain the fit shown in Figure 2b with T bath = (7 ± .5) K. From T bath we thus obtain the value of C M q = 2.5 ± 0.3 for the data presented in Figure 2d. The thermal noise contribution is shown as the blue area and can easily be found using Eq. (S1).
For the two phase quadrature data sets shown the mechanical oscillator was optically broadened to γ M = 2π × 2.7 kHz. This required the input LO 2 power to be adjusted as the cavity half bandwidth and single photon coupling rate varied slightly between runs. This was due to a varying membrane position with respect to the intra-cavity field on different days caused by overnight temperature cycles. The experimentally realised parameters for these are displayed in Table I.

C. Hybrid system
A detailed schematic of the experimental setup is shown in Fig. S1: a travelling light field interacts with the atomic spin and the mechanical oscillator in a cascaded way. The driving beams for the system, LO 1 and LO 2 , as well as the local oscillator LO 3 for the homodyne measurement are generated by a Ti-Sapphire laser. The light is shot noise limited for the relevant powers and Fourier frequencies of interest in the quadratures that matter for both systems.
The spin-mechanics interface requires filtering of the spins' output field. The atomic spins respond to modulation out of phase and in the orthogonal polarization to its local oscillator (denoted by the quantum field operatorX L,in in the main text). The mechanical oscillator responds to modulation in phase and in the same polarization mode as its own driving local oscillator (represented by the field operatorX M L,in ). Therefore, LO 2 should be in an orthogonal polarization mode and π/2 phase shifted with respect to LO1. Two filtering stages are required, one for polarization and one for phase, both of them being depicted in Fig. S1 and described in the following. The polarization filtering is done using HWP 1 and PBS 1 right after the microcell, decoupling LO 1 from the quantum fluctuations of interest in the orthogonal polarization quadrature; the phase filtering is realized using a Mach-Zehnder interferometer with output at PBS 2 , setting a variable phase for the spin sideband fields {X S L,out ,P S L,out } in respect to LO 2 . The driving local oscillator and the sidebands are then projected in the same polarization mode with HWP 3 and PBS 3 and coupled to the optomechanical cavity. To detect the optical quadrature of interest, balanced polarimetry with the local oscillator LO 3 (with the aid of another Mach-Zehnder interferometer) is performed when the phase quadrature is of interest; to measure of the amplitude quadrature, the local oscillator is removed and all light is directed to a single photodiode.
In the experiment, PBS 1 extinguishes LO 1 better than 1:10 3 from the optical path with little loss of the modulation sidebands or those carrying information about the spin oscillator.
[ [ [ FIG. S1. Detailed schematic of the experimental setup. The atomic spin system is pictured in the black-brown dashed box, along with its B-field and optical pumping; in the blue lined box, the optomechanical membrane-in-the-middle setup. The hybrid system is probed via a travelling optical mode. The atomic system, driven by LO1 with linear polarization angle set by 0 has its output polarization filtered in 1 and is recombined with the correct phase with LO2 in 2 , set electronically via suitable detection in D1; 3 ensures that both local oscillator and the filtered atomic response have the same polarization. The optomechanical system is probed in reflection and frequency stabilized via PDH locking in the unused port. Phase sensitive detection is done via homodyning with LO3 in D2.
An electro-optic modulator (EOM) in LO 1 is used for locking the phase filtering interferometer. The phase and axis of the EOM are adjusted in such a way so that a voltage modulation (small with respect to the π voltage) results in a small modulation predominantly in the degree of circular polarization of light (quantified by theŜ z Stokes component). The ratio of circular polarization modulation to linear polarization modulation introduced by the EOM is typically ∼ 10 5 in power. Sinusoidal modulation sidebands at 10MHz, far from both oscillators' responses, provide the phase reference. These sidebands are combined with LO 2 in the output of the interferometer, PBS 2 . A half-waveplate allows for an adjustable fraction of the sideband power to be used for locking (typically ∼ 5%). The demodulated result of the balanced polarimetry measurement of the locking signal in D 1 is proportional to cos δφ LO1,2 , where δφ LO1,2 is the phase difference between LO 1 and LO 2 . Feedback on a piezoelectric transducer proportional to this signal allows us to lock the new local oscillator in phase with the sideband quadrature that drove the spin oscillator.
The half-waveplate HWP 2 and PBS 2 project a small portion of LO 2 (∼ 3%) and most of the sideband signal (∼ 97%) into the same polarization mode. Suitable optics direct the beam onto the optomechanical cavity with a total optical power transmission for the spin system response sidebands in the 60 − 65% range. These sidebands are in the same spatial mode as LO 1 , which is modematched to the cavity with an efficiency of η mm 90%. The modematching is defined as the fraction of incident LO 1 power going into the TEM 00 compared to all TEM modes.
When the characterization of the atomic spin oscillator is performed, a function generator provides a white noise (WN) modulation over the interesting frequency range, from 1MHz to 2MHz; typical values for the added modulated WN photons (in units of SN) ranges from 0.5 to 100.
The spectra ofP L,out are measured by balanced homodyning of the field reflected off the optomechanical cavity with LO 3 , with power in the order of 1.5 mW, which is locked to the DC zero of the interference fringe with LO 2 , thus ensuring that the phase quadrature is being measured.
The model fits and knowledge of all relevant system parameters provide a reliable reference point from which we calibrate the spectra in units of the mechanical zero point fluctuations. For example, the right vertical axis in Fig. 1d shows the spectral density of motion in units of x 2 zpf /kHz, calibrated by dividing the thermal noise by n th . Integrating the power spectral density data we find the observed membrane variance of 11.6 × x 2 zpf . Subtracting the thermal noise variance 4.3 × x 2 zpf we obtain the QBA variance 7.3 × x 2 zpf . Losses in the system are due to the finite transmission coefficient between the spin and the optomechanical systems of η 1 = 0.53, which includes the finite optomechanical coupling efficiency 0.88, and the transmission coefficient between the optomechanical system and the detection of η 2 = 0.64 which includes the quantum efficiency of the photodetector of 0.90. These values vary within a few percentage points from experiment to experiment.

D. Optomechanical System
The optomechanical system is described by the standard linearized Hamiltonian where [X M ,P M ] = i are the dimensionless canonical operators for the mechanical system, and [â,â † ] = 1 are annihilation/creation operators for cavity photons. ∆ = ω L − ω c is the detuning of the driving laser from the cavity resonance ω c . The linearized optomechanical coupling rate g = g 0 |α| depends on the single photon coupling rate g 0 of the optomechanical system and the intracavity amplitude α. It is linked to the optomechanical readout rate Γ M introduced in the main text by where κ is the cavity half linewidth. The phase φ = arctan(∆/κ) denotes the phase of the intracavity field amplitude α relative to driving field, as is discussed further below in Eq. (S32). Here we take the incoming field as the phase reference instead of the intracavity field (as is usually done in cavity optomechanics) since we eventually interested in the transfer matrix for the incoming/outgoing amplitudes resulting from this Hamiltonian. Including decay and Langevin noise forces the equations of motion corresponding to the Hamiltonian arė , whereâ in (t) andv in (t) are incoming quantum fields driving the cavity through port 1 and 2, respectively, cf. Fig. S2; . The partial decay rates κ 1(2) fulfill κ = κ 1 + κ 2 . The linewidth of the mechanical resonance (excluding optical broadening) is γ M 0 , and the thermal Langenvin force isf (t). In the high temperature limit we can take f (t)f (t ) = δ(t − t )(n + 1/2) wheren k B T bath /( Ω M ).
FIG. S2. Schematic of the setup: The atomic spin is driven by light noise XL,in and spin noise FS. Output light of the the spin system X S L,out is channeled to the atomic system, and experiences losses characterized by a transmissivity η1 associated with additional light noise V1,in and a phase rotation by an angle ϕ, resulting in a driving field X M L,in of the optomechanical system. The optomechanical cavity is two-sided with decay rates κ1 and κ2. The optomechanical system is driven in addition by light noise Vin and a thermal force F. The output field of the optomechanical system experiences further losses with transmissivity η2 associated with additional light noise V2,in.
In the frequency domain the equations of motion read We define field quadratures aŝ and similar definitions are used for quadratures of incoming/outgoing fields. In terms of these the equations of motion in frequency domain are which can be conveniently written in terms of block matrices The equations of motion are solved by where the inverse Block matrix can be expressed in two equivalent forms by means of the Schur complements The effective mechanical susceptibility including optically induced shift and broadening is The intracavity quadratures following from Eqs. (S5) and (S6) arê , and the field reflected off the cavity in port 1 iŝ The optomechanical transfer matrix M(Ω) is explicitly given by In the form given in the second line the dependence on the effective mechanical susceptibility becomes evident.
We note that for a broadband cavity (κ ∆, Ω M , Ω) and neglecting losses (κ 2 = 0) one recovers from Eq. (S10) the simple optomechanical input-output relation stated in the main text, . (S12) In the limit considered here the susceptibility corresponds to the one of the bare mechanical system (without shift and broadening). For nonzero detuning and taking into account effects of a finite cavity linewidth, the more involved input-output relations described by Eqs. (S10) have to be considered in general. However, in the unresolved-sideband regime (κ Ω M , Ω) we may obtain a simplified expression for the optomechanical transfer matrix (S11). To this end, we note that the cavity response to the individual sideband components at ±Ω of the light quadratures (S4) is governed by the complex Lorentzian where we have introduced its polar decomposition. In terms of this, Eq. (S11) can be reexpressed as (again neglecting cavity losses for simplicity, κ 2 = 0) where φ = θ(0) (see discussion of Eq. (S32) below) and χ M (Ω) is the effective mechanical susceptibility (S9). To obtain a simpler expression for Eq. (S14) in the regime κ Ω M , Ω, we expand |L(Ω)| and θ(Ω) to linear order around the carrier frequency (Ω = 0 in the rotating frame), resulting in the optomechanical scattering matrix to leading order in δθ, δL (the phase prefactor does not affect the resulting spectra and will be suppressed for brevity henceforth). The transfer matrix in Eq. (S17) interpolates between the simple result in Eq. (S12), which is valid in the limit κ → ∞, and the general result in Eq. (S11).
When considering the field reflected off the cavity in port 1 in Eq. (S10), the finite modematching η mm of the input quadraturesX L,in (Ω) to the cavity quadraturesX L (Ω) is treated as equivalent to the input port having higher loss, i.e. κ 1 → η mm κ 1 . The total cavity loss remains fixed κ = κ 1 + κ 2 and we simply treat κ 2 → κ 2 + (1 − η mm )κ 1 as the input for the additional vacuum noise.

E. Atomic Spins System
As discussed in Ref. [S2, S6], in the limit of low saturation and large detuning from the atomic resonance, the Hamiltonian affecting the atomic spin and light polarization observables can be written in the form: whereĴ z is the dimensionless ( = 1) collective spin component along the direction of light propagation (taken here to coincide with the z axis in the lab frame) andŜ z is the Stokes component of light that measures the degree of circular polarization. The parameter α depends on the detuning from the resonance ∆ S , on the area A of interaction and on physical constants: where Γ sp is the spontaneous emission rate associated with the optical transition, λ is the wavelength of light and α 1 (∆) is a numerical factor that depends on the specific atomic structure and for detunings much larger than the excited state hyperfine structure can be approximated to be unity.
For an ensemble of a large number of atoms, highly polarized along the direction of a static magnetic field (x direction), the Holstein-Primakoff transformation can be performed and map the collective spin operators to position and momentum operators of an effective (spin) oscillator: Here we represent the macroscopic mean polarization by its x-projection J x = Ĵ x (including its sign) rather than merely its magnitude (as done in the main text for simplicity). As described in the main text and above in the Methods section, the relative sign of J x and B can be either positive or negative, reflecting whether the macroscopic spin is aligned or anti-aligned with respect to the applied magnetic field. For the case of this work, where Cesium atoms are polarized in the F = 4 hyperfine manifold of the ground electronic state, positive sgn(J x /B) corresponds to a negative mass oscillator (energy should be extracted to remove the ensemble from the fully polarized state), whereas negative sgn(J x /B) corresponds to a positive mass oscillator. The presence of the static magnetic field adds the Hamiltonian term with µ B being the Bohr magneton and g F = 1/4 the Landé factor for the F = 4 manifold. In the language of spin oscillators this Hamiltonian term affects the evolution ofX S andP S in the following way: where Ω S = −sgn(J x )µ B g F B so that Ω S > 0(< 0) refers to the positive (negative) mass scenario.
A similar mapping can be performed with the Stokes components of light. For linearly polarized light in the x direction with Stokes component S x = sgn(S x )Φ/2, where Φ is the photon flux, the mapping can be written in the form:X From Eqs. (S18), (S20), (S21) (S23) and (S27) we can write the Hamiltonian aŝ The input-output relationships for the Stokes componentŝ are mapped into:X where the readout rate Γ S is The atomic spin dynamics, including the effects of its interaction with a Markovian reservoir, is withF S (t) being the random Langevin force acting on the spin; this force is the analogous of the thermal noisef that acts on the mechanical oscillator. Its correlation function is F S (t)F S (t ) = δ(t − t )(n S + 1/2), where a thermal spin occupancy n S > 0 reflects the excess noise induced by imperfect polarization of the ensemble. In the above analysis, the effect of tensor polarizability in the evolution of the light and spin state was neglected. For the detuning used in the experiment (∆ S ∼ 3GHz) the effect of the tensor polarizability is estimated to be on the few percent level [S12].
In frequency space, the spin system is structurally identical to the one of the simple limit considered in Eq. (S12) for the optomechanical system, that is, Fourier transforming and solving Eqs. (S26,S28,S29), one obtains the matrix relationship X S L,out (Ω) = S(Ω)X L,in (Ω) + Γ S γ S χ S (Ω)F S (Ω).
where the spin oscillator susceptibility is χ S (Ω) = 2Ω S /(Ω 2 S − Ω 2 − 2iΩγ S ). The spin thermal noise is represented by F S (Ω) = [0,F S (Ω)] T . Here we adopt a phenomenological model for the susceptibility of the spin oscillator. A microscopic derivation along the lines of [S13] would result in a slightly different susceptibility with corrections to the present one scaling as Q −1 S where Q S 1 is the quality factor of the atomic oscillator.

F. Hybrid System
The two systems are connected such that X M L,in (Ω) = X M S,out (Ω), as shown schematically in Fig. S2. Taking into account losses and further phase shifts as indicated in the figure the compound transfer matrix for the hybrid optomechanical-spin system iŝ where η 1 and η 2 denote the transmission efficiencies from the spin system to the optomechanical cavity and from the optomechanical cavity to the detector, respectively. Vacuum noises incurred through these losses are described bŷ V 1(2),in (Ω). An optional phase shift ϕ introduced deliberately in between the two systems is accounted for by the rotation matrix O ϕ . Finally, the homodyne detection is performed in the frame of the classical field after the optomechanical system where it has acquired a phase shift relative to the field before the optomechanical cavity. This phase is found as follows: The classical intracavity amplitude α is connected to the incoming amplitude α in by where κ = κ 1 + κ 2 . The outgoing field is where ψ = arctan(∆/(κ 1 − κ 2 )). Accordingly, the measured field quadratureP L,meas is determined by This relation is used to determine the measured noise spectral densities shown in the main text. For simplicity of notation, the measured quadratureP L,meas is referred to asP L,out in the main text and other parts of the Methods.
We will now use the transfer matrix of the hybrid system to analyse the QBA contribution to the optical output field [Eq. (S31), 1st line]. For the case when the opto-mechanical damping dominates the membrane response, γ M γ M 0 , and in the sideband unresolved limit, Ω M κ, we can apply the approximate optomechanical scattering matrix (S17) to find (ignoring optical losses η 1 = 0 = η 2 , κ 2 = 0 for simplicity and setting ϕ = 0) where L 0 is the empty cavity Lorentzian response and δL is the difference in cavity response at frequencies ±Ω κ. Only with δL = 0 (LO 2 tuned to cavity resonance) the spin QBA and the mechanical QBA add/subtract inP L,meas . To characterize the quantum cooperativity, C S q , and the readout rate, Γ S , one can use the fact that a single light quadrature is coupled to the oscillator: by suitable modulation ofX L,in one can boost the contribution of the measurement-induced back action.
As thoroughly discussed in the previous sections and summarized by equation (S30), the input-output relations for the continuous readout of a harmonic oscillator are X L,out P L,out = X L,in P L,in + Γv T 1 Lv 2 0 0 1 0 in which Γ is the readout rate, γ the decay rate and In a more straightforward language, equation (S35) becomes X L,out P L,out = X L,in P L,in + R BA 0 0 1 0 with R BA and R Th being the response functions of the oscillator to the back action and thermal forces. The effect of losses is also important, as there is an admixture of uncorrelated vacuum, indicated by the subscript v, with the signal of interest; therefore X L,out P L,out → √ η X L,out P L,out The PSD for both light quadratures are calculated from the absolute square of the equation (S36): Therefore, it is explicit that to boost the back action component of the oscillator readout in comparison to the other noise contributions, one should modulate the in-phase quadrature of light,X L . Doing so, the input spectral densities are X L,in X † L,in → (n W N + 1) X L,in X † L,in , and X L,i (ω)X † L,j (−ω ) = δ(ω − ω )δ ij , in which i, j represent the different sources of fluctuations. Therefore, the input-output relations from (S38) are S XX = ηn W N + 1 S P P = η R 2 BA (n W N + 1) + R 2 Th + 1.
Experimentally, to be able to calculate the back action to thermal noise ratio, one needs to measure the (i) response of the system to SN drive and (ii) the response of the system with some known modulation n W N , for a given probe power. Calibrating the curves in shot noise units, the measured spectral on-resonance heights after the subtraction of the white noise contribution, defined here as A and B, are S W N P P − 1 = B = η R 2 BA (n W N + 1) + R 2 Th S SN P P − 1 = A = η R 2 BA + R 2 Th , therefore, the desired ratio is: This technique was used to calibrate C S q , the quantum cooperativity of the spin oscillator. For this measurement, η = 0.7 is the detection efficiency, S WN is the spectral density of added white noise in units of vacuum noise and S QBA and S TH are the back action and thermal spectral densities, respectively. The measurements of the phase noise presented in (Figure 2c) and of S WN are performed with polarization interferometry using LO 1 calibrated to the shot noise of LO 1 . From the phase noise S PP,0 and S PP,N measured for n WN = 0 (vacuum input) and S WN = 1.